{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 简介"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-25T08:41:01.220832Z",
     "start_time": "2019-04-25T08:41:01.215818Z"
    }
   },
   "source": [
    "交叉验证（Cross Validation）是常用的一种用来评估模型效果的方法。\n",
    "\n",
    "当样本分布发生变化时，交叉验证无法准确评估模型在测试集上的效果，这导致模型在测试集上的效果远低于训练集。\n",
    "\n",
    "通过本文，你将通过一个kaggle的比赛实例了解到，样本分布变化如何影响建模，如何通过对抗验证辨别样本的分布变化，以及有哪些应对方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 什么是「样本分布变化」？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-24T14:20:35.857616Z",
     "start_time": "2019-04-24T14:20:35.846925Z"
    }
   },
   "source": [
    "在真实的业务场景中，我们经常会遇到「样本分布变化」的问题。\n",
    "\n",
    "主要体现在训练集和测试集的分布存在的差异。比如，在化妆品或者医美市场，男性的比例越来越多。基于过去的数据构建的模型，渐渐不适用于现在。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 为什么「样本分布变化」的时候，交叉验证不适用？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当我们要做一个模型，来预测人们在超市的消费习惯。\n",
    "\n",
    "我们的训练样本主要是18岁-25岁的年轻人构成，而测试样本主要是70岁以上的老人组成。这时样本分布就发生了变化。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-26T03:27:19.165668Z",
     "start_time": "2019-04-26T03:27:19.050902Z"
    }
   },
   "source": [
    "![样本分布变化](./images/Change_in_Distribution.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这种情况下，使用交叉验证，其实无法准确评估模型的效果。原因是，交叉验证的验证集和测试集不够相似。\n",
    "\n",
    "交叉验证中，每一折的验证集都是从训练集随机抽取的。随机抽取的验证集的分布和整体的训练集是相同的，也就意味着每一折的验证集都和测试集的分布存在较大的差异。\n",
    "\n",
    "所以在样本分布变化时，通过交叉验证的方式构建的模型，在测试集上的表现相较于训练集通常会打折扣。稍后我们会通过一个实例来确认这一点。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-25T09:49:50.918672Z",
     "start_time": "2019-04-25T09:49:50.915679Z"
    }
   },
   "source": [
    "# 什么是对抗验证（Adversarial Validation）？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-19T09:25:34.263897Z",
     "start_time": "2019-04-19T09:25:34.257934Z"
    }
   },
   "source": [
    "[对抗验证（Adversarial Validation）](http://fastml.com/adversarial-validation-part-one/)，并不像交叉验证是一种评估模型效果的方法，而是一种用来确认训练集和测试集的分布是否变化的方法。\n",
    "\n",
    "它的本质是，构造一个分类模型，来预测样本是训练集或测试集的概率。\n",
    "\n",
    "如果这个模型的效果不错（通常来说AUC在0.7以上），那么可以说明我们的训练集和测试集存在较大的差异。\n",
    "\n",
    "如下图，仍然以「预测人们在超市的消费习惯」为例。因为训练集主要是18岁-25岁的年轻人，测试集主要是70岁以上的老人，那么通过「年龄」，我们就能够比较好的区分出训练集和测试集。\n",
    "\n",
    "<img src=\"./images/Adversarial_Validation.png\" width=\"500\" height=\"500\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-19T09:08:25.769307Z",
     "start_time": "2019-04-19T09:08:25.763303Z"
    }
   },
   "source": [
    "具体步骤如下：\n",
    "\n",
    "* 定义新的Y（因变量）：样本是train还是test。训练集中的样本统一标记为0，测试集则标记为1。\n",
    "* 将 Train 和 Test 合成一个数据集\n",
    "* 构造一个模型，拟合新定义的Y。\n",
    "* 观察模型效果：如果模型的AUC超过0.7，说明了 Train 和 Test 的分布存在较大的差异。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 分布变化时，优于交叉验证的方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "主要是三种方法：\n",
    "\n",
    "* 人工划分验证集\n",
    "* 和测试集最相似的样本作为验证集\n",
    "* 有权重的交叉验证"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 人工划分验证集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "人工划分验证集，需要我们对数据有充分的了解。\n",
    "\n",
    "因为我知道这次比赛的数据是根据时间划分的，所以我的验证集同样可以根据时间划分。\n",
    "\n",
    "如果我们不清楚训练集和测试集如何划分，可以采用后面两种方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 用和测试集分布最相似的样本，作为验证集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果对数据没有充分了解，如何找到训练集中，和测试集分布最相似的样本呢？\n",
    "\n",
    "这就会用到我们做对抗验证时，模型预测样本是测试集的概率。概率越高，则说明和测试集越相似。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 有权重的交叉验证"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不仅可以用对抗验证中，模型预测样本是测试集的概率来划分验证集，也可以将这个概率作为样本的权重。\n",
    "\n",
    "概率越高，和测试集就越相似，权重就越高。\n",
    "\n",
    "这样，我们就可以做有权重的交叉验证。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-25T00:46:22.654844Z",
     "start_time": "2019-04-25T00:46:22.644842Z"
    }
   },
   "source": [
    "# 实例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-25T00:46:44.650549Z",
     "start_time": "2019-04-25T00:46:44.641005Z"
    }
   },
   "source": [
    "![Microsoft Malware Prediction](./images/Microsoft_Malware_Competition.png)\n",
    "\n",
    "\n",
    "这里用到的数据来自Kaggle上的[微软恶意软件比赛](https://www.kaggle.com/c/microsoft-malware-prediction/overview)。\n",
    "\n",
    "每一个样本代表着一台电脑。这次比赛的目标是：预测电脑受到恶意软件攻击的概率。\n",
    "\n",
    "因为这次比赛的 Train 和 Test 是根据时间划分的，所以Train 和 Test 的分布非常不同，很具有代表性。\n",
    "\n",
    "如果需要数据，可以从[Kaggle](https://www.kaggle.com/c/microsoft-malware-prediction/data)下载。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:22:27.815383Z",
     "start_time": "2019-04-29T14:22:26.717639Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from tqdm import tqdm\n",
    "import lightgbm as lgb\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.model_selection import KFold\n",
    "\n",
    "# Memory management\n",
    "import gc\n",
    "gc.enable()\n",
    "\n",
    "# Plot\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('ggplot')\n",
    "\n",
    "# Suppress warnings\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:22:27.851539Z",
     "start_time": "2019-04-29T14:22:27.842588Z"
    }
   },
   "outputs": [],
   "source": [
    "# dtypes = {\n",
    "#     'MachineIdentifier':                                    'category',\n",
    "#     'ProductName':                                          'category',\n",
    "#     'EngineVersion':                                        'category',\n",
    "#     'AppVersion':                                           'category',\n",
    "#     'AvSigVersion':                                         'category',\n",
    "#     'IsBeta':                                               'int8',\n",
    "#     'RtpStateBitfield':                                     'float16',\n",
    "#     'IsSxsPassiveMode':                                     'int8',\n",
    "#     'DefaultBrowsersIdentifier':                            'float32',\n",
    "#     'AVProductStatesIdentifier':                            'float32',\n",
    "#     'AVProductsInstalled':                                  'float16',\n",
    "#     'AVProductsEnabled':                                    'float16',\n",
    "#     'HasTpm':                                               'int8',\n",
    "#     'CountryIdentifier':                                    'int16',\n",
    "#     'CityIdentifier':                                       'float32',\n",
    "#     'OrganizationIdentifier':                               'float16',\n",
    "#     'GeoNameIdentifier':                                    'float16',\n",
    "#     'LocaleEnglishNameIdentifier':                          'int16',\n",
    "#     'Platform':                                             'category',\n",
    "#     'Processor':                                            'category',\n",
    "#     'OsVer':                                                'category',\n",
    "#     'OsBuild':                                              'int16',\n",
    "#     'OsSuite':                                              'int16',\n",
    "#     'OsPlatformSubRelease':                                 'category',\n",
    "#     'OsBuildLab':                                           'category',\n",
    "#     'SkuEdition':                                           'category',\n",
    "#     'IsProtected':                                          'float16',\n",
    "#     'AutoSampleOptIn':                                      'int8',\n",
    "#     'PuaMode':                                              'category',\n",
    "#     'SMode':                                                'float16',\n",
    "#     'IeVerIdentifier':                                      'float16',\n",
    "#     'SmartScreen':                                          'category',\n",
    "#     'Firewall':                                             'float16',\n",
    "#     'UacLuaenable':                                         'float64',  # was 'float32'\n",
    "#     'Census_MDC2FormFactor':                                'category',\n",
    "#     'Census_DeviceFamily':                                  'category',\n",
    "#     'Census_OEMNameIdentifier':                             'float32',  # was 'float16'\n",
    "#     'Census_OEMModelIdentifier':                            'float32',\n",
    "#     'Census_ProcessorCoreCount':                            'float16',\n",
    "#     'Census_ProcessorManufacturerIdentifier':               'float16',\n",
    "#     'Census_ProcessorModelIdentifier':                      'float32',  # was 'float16'\n",
    "#     'Census_ProcessorClass':                                'category',\n",
    "#     'Census_PrimaryDiskTotalCapacity':                      'float64',  # was 'float32'\n",
    "#     'Census_PrimaryDiskTypeName':                           'category',\n",
    "#     'Census_SystemVolumeTotalCapacity':                     'float64',  # was 'float32'\n",
    "#     'Census_HasOpticalDiskDrive':                           'int8',\n",
    "#     'Census_TotalPhysicalRAM':                              'float32',\n",
    "#     'Census_ChassisTypeName':                               'category',\n",
    "#     'Census_InternalPrimaryDiagonalDisplaySizeInInches':    'float32',  # was 'float16'\n",
    "#     'Census_InternalPrimaryDisplayResolutionHorizontal':    'float32',  # was 'float16'\n",
    "#     'Census_InternalPrimaryDisplayResolutionVertical':      'float32',  # was 'float16'\n",
    "#     'Census_PowerPlatformRoleName':                         'category',\n",
    "#     'Census_InternalBatteryType':                           'category',\n",
    "#     'Census_InternalBatteryNumberOfCharges':                'float64',  # was 'float32'\n",
    "#     'Census_OSVersion':                                     'category',\n",
    "#     'Census_OSArchitecture':                                'category',\n",
    "#     'Census_OSBranch':                                      'category',\n",
    "#     'Census_OSBuildNumber':                                 'int16',\n",
    "#     'Census_OSBuildRevision':                               'int32',\n",
    "#     'Census_OSEdition':                                     'category',\n",
    "#     'Census_OSSkuName':                                     'category',\n",
    "#     'Census_OSInstallTypeName':                             'category',\n",
    "#     'Census_OSInstallLanguageIdentifier':                   'float16',\n",
    "#     'Census_OSUILocaleIdentifier':                          'int16',\n",
    "#     'Census_OSWUAutoUpdateOptionsName':                     'category',\n",
    "#     'Census_IsPortableOperatingSystem':                     'int8',\n",
    "#     'Census_GenuineStateName':                              'category',\n",
    "#     'Census_ActivationChannel':                             'category',\n",
    "#     'Census_IsFlightingInternal':                           'float16',\n",
    "#     'Census_IsFlightsDisabled':                             'float16',\n",
    "#     'Census_FlightRing':                                    'category',\n",
    "#     'Census_ThresholdOptIn':                                'float16',\n",
    "#     'Census_FirmwareManufacturerIdentifier':                'float16',\n",
    "#     'Census_FirmwareVersionIdentifier':                     'float32',\n",
    "#     'Census_IsSecureBootEnabled':                           'int8',\n",
    "#     'Census_IsWIMBootEnabled':                              'float16',\n",
    "#     'Census_IsVirtualDevice':                               'float16',\n",
    "#     'Census_IsTouchEnabled':                                'int8',\n",
    "#     'Census_IsPenCapable':                                  'int8',\n",
    "#     'Census_IsAlwaysOnAlwaysConnectedCapable':              'float16',\n",
    "#     'Wdft_IsGamer':                                         'float16',\n",
    "#     'Wdft_RegionIdentifier':                                'float16',\n",
    "#     'HasDetections':                                        'int8'\n",
    "# }\n",
    "\n",
    "# df_all = pd.read_csv('./input/train.csv.zip', dtype=dtypes) \n",
    "## 对训练集随机抽取2%的样本\n",
    "# df_all = df_all.sample(frac=0.02, random_state=123)\n",
    "# df_all.to_csv('./input/train_sample.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为这次比赛的训练集有800万样本，测试集有700万样本。为了方便演示，这里我仅随机抽取训练集中2%的样本，而且不使用测试集的数据。我们稍后将从训练集中拆分一个数据集，作为我们的测试集。\n",
    "\n",
    "不使用这次比赛原本的测试集可以节省很多时间。因为测试集有700万的样本，每做一次预测会消耗大量时间。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:22:30.544411Z",
     "start_time": "2019-04-29T14:22:28.104648Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MachineIdentifier</th>\n",
       "      <th>ProductName</th>\n",
       "      <th>EngineVersion</th>\n",
       "      <th>AppVersion</th>\n",
       "      <th>AvSigVersion</th>\n",
       "      <th>IsBeta</th>\n",
       "      <th>RtpStateBitfield</th>\n",
       "      <th>IsSxsPassiveMode</th>\n",
       "      <th>DefaultBrowsersIdentifier</th>\n",
       "      <th>AVProductStatesIdentifier</th>\n",
       "      <th>...</th>\n",
       "      <th>Census_FirmwareVersionIdentifier</th>\n",
       "      <th>Census_IsSecureBootEnabled</th>\n",
       "      <th>Census_IsWIMBootEnabled</th>\n",
       "      <th>Census_IsVirtualDevice</th>\n",
       "      <th>Census_IsTouchEnabled</th>\n",
       "      <th>Census_IsPenCapable</th>\n",
       "      <th>Census_IsAlwaysOnAlwaysConnectedCapable</th>\n",
       "      <th>Wdft_IsGamer</th>\n",
       "      <th>Wdft_RegionIdentifier</th>\n",
       "      <th>HasDetections</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>586d40804b950d0376575fdf10ee89ae</td>\n",
       "      <td>win8defender</td>\n",
       "      <td>1.1.15100.1</td>\n",
       "      <td>4.18.1806.18062</td>\n",
       "      <td>1.273.520.0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>53447.0</td>\n",
       "      <td>...</td>\n",
       "      <td>27767.0</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>65fb3fae2d37f90e6b3174592f2490a8</td>\n",
       "      <td>win8defender</td>\n",
       "      <td>1.1.15200.1</td>\n",
       "      <td>4.18.1807.18075</td>\n",
       "      <td>1.275.453.0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7945.0</td>\n",
       "      <td>...</td>\n",
       "      <td>14353.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>c23aa37fb69e00afe2668ed150dee1ea</td>\n",
       "      <td>win8defender</td>\n",
       "      <td>1.1.15100.1</td>\n",
       "      <td>4.18.1807.18075</td>\n",
       "      <td>1.273.689.0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>53447.0</td>\n",
       "      <td>...</td>\n",
       "      <td>8941.0</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>cba75d6c4d9b6533591e94b9cb8a5df5</td>\n",
       "      <td>win8defender</td>\n",
       "      <td>1.1.15200.1</td>\n",
       "      <td>4.12.16299.15</td>\n",
       "      <td>1.275.483.0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>68585.0</td>\n",
       "      <td>...</td>\n",
       "      <td>46589.0</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>149746364c6b763662d03e1f263029fd</td>\n",
       "      <td>win8defender</td>\n",
       "      <td>1.1.15200.1</td>\n",
       "      <td>4.18.1807.18075</td>\n",
       "      <td>1.275.215.0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>53447.0</td>\n",
       "      <td>...</td>\n",
       "      <td>52530.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 83 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                  MachineIdentifier   ProductName EngineVersion  \\\n",
       "0  586d40804b950d0376575fdf10ee89ae  win8defender   1.1.15100.1   \n",
       "1  65fb3fae2d37f90e6b3174592f2490a8  win8defender   1.1.15200.1   \n",
       "2  c23aa37fb69e00afe2668ed150dee1ea  win8defender   1.1.15100.1   \n",
       "3  cba75d6c4d9b6533591e94b9cb8a5df5  win8defender   1.1.15200.1   \n",
       "4  149746364c6b763662d03e1f263029fd  win8defender   1.1.15200.1   \n",
       "\n",
       "        AppVersion AvSigVersion  IsBeta  RtpStateBitfield  IsSxsPassiveMode  \\\n",
       "0  4.18.1806.18062  1.273.520.0       0               7.0                 0   \n",
       "1  4.18.1807.18075  1.275.453.0       0               7.0                 0   \n",
       "2  4.18.1807.18075  1.273.689.0       0               7.0                 0   \n",
       "3    4.12.16299.15  1.275.483.0       0               7.0                 0   \n",
       "4  4.18.1807.18075  1.275.215.0       0               7.0                 0   \n",
       "\n",
       "   DefaultBrowsersIdentifier  AVProductStatesIdentifier      ...       \\\n",
       "0                        NaN                    53447.0      ...        \n",
       "1                        NaN                     7945.0      ...        \n",
       "2                        NaN                    53447.0      ...        \n",
       "3                        NaN                    68585.0      ...        \n",
       "4                        NaN                    53447.0      ...        \n",
       "\n",
       "   Census_FirmwareVersionIdentifier  Census_IsSecureBootEnabled  \\\n",
       "0                           27767.0                           1   \n",
       "1                           14353.0                           0   \n",
       "2                            8941.0                           1   \n",
       "3                           46589.0                           1   \n",
       "4                           52530.0                           0   \n",
       "\n",
       "   Census_IsWIMBootEnabled  Census_IsVirtualDevice  Census_IsTouchEnabled  \\\n",
       "0                      NaN                     0.0                      0   \n",
       "1                      NaN                     0.0                      0   \n",
       "2                      NaN                     0.0                      0   \n",
       "3                      NaN                     0.0                      0   \n",
       "4                      NaN                     0.0                      0   \n",
       "\n",
       "   Census_IsPenCapable  Census_IsAlwaysOnAlwaysConnectedCapable  Wdft_IsGamer  \\\n",
       "0                    0                                      0.0           1.0   \n",
       "1                    0                                      0.0           0.0   \n",
       "2                    0                                      0.0           1.0   \n",
       "3                    0                                      0.0           1.0   \n",
       "4                    0                                      0.0           NaN   \n",
       "\n",
       "  Wdft_RegionIdentifier HasDetections  \n",
       "0                  15.0             1  \n",
       "1                  10.0             0  \n",
       "2                   1.0             1  \n",
       "3                   7.0             1  \n",
       "4                   NaN             0  \n",
       "\n",
       "[5 rows x 83 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读取训练集中随机抽取的2%的样本\n",
    "df_all = pd.read_csv('./input/train_sample.csv') \n",
    "df_all.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本次比赛的数据中提供了电脑的 Windows Defender（Windows系统自带的杀毒软件）版本号，所以我们可以通过该本版号发布的时间，粗略的推测采集该样本的时间。\n",
    "\n",
    "这里AvSigVersionTimestamps就是各个Windows Defender版本对应的发布时间。\n",
    "\n",
    "通过和该数据匹配，我们生成了一个新的字段 -- Date（日期）。这个字段稍后会起作用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:22:30.945542Z",
     "start_time": "2019-04-29T14:22:30.777409Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 读取Windows Defender版本对应的发布时间\n",
    "datedict = np.load('./input/AvSigVersionTimestamps.npy')\n",
    "datedict = datedict[()]\n",
    "\n",
    "# 生成新的变量Date\n",
    "df_all['Date'] = df_all['AvSigVersion'].map(datedict)\n",
    "\n",
    "# MachineIdentifier是每台电脑的唯一识别号，对于模型的预测没有任何帮助，所以剔除。\n",
    "df_all.drop(['MachineIdentifier'], axis=1, inplace=True) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据清理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 去掉无意义变量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里无意义变量的定义是：变量的某个值（可以是空值）的占比大于99%。\n",
    "\n",
    "比如，如果所有样本的「系统版本」都是Win7，那么「系统版本」这个变量就没有意义。\n",
    "\n",
    "所以，如果一个变量，99%以上的样本，都是一个值，那么这个变量接近于无意义。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:22:32.625278Z",
     "start_time": "2019-04-29T14:22:31.190681Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data Shape:  (178430, 75)\n",
      "['IsBeta', 'AutoSampleOptIn', 'PuaMode', 'UacLuaenable', 'Census_DeviceFamily', 'Census_ProcessorClass', 'Census_IsPortableOperatingSystem', 'Census_IsVirtualDevice']\n"
     ]
    }
   ],
   "source": [
    "bad_cols = []\n",
    "for col in df_all.columns:\n",
    "    rate_train = df_all[col].value_counts(normalize=True, dropna=False).values[0]\n",
    "    if rate_train > 0.99:\n",
    "        bad_cols.append(col)\n",
    "\n",
    "df_all = df_all.drop(bad_cols, axis=1)\n",
    "\n",
    "print('Data Shape: ', df_all.shape)\n",
    "print(bad_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 定义数据类型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里是通过EDA(Exploratory Data Analysis)的方式，人工判断的变量类型。\n",
    "\n",
    "总共将变量分为\n",
    "* 数值变量（true_numerical_columns）\n",
    "* 一般的分类变量（categorical_columns）\n",
    "* 类别非常多的分类变量（categorical_columns_high_car）：比如中国的城市（北京、上海、深圳、重庆等等等...）\n",
    "\n",
    "如果你对这次比赛的细节感兴趣，可以再深入研究为什么这样判断。这里就不详细阐述原因了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:22:32.911247Z",
     "start_time": "2019-04-29T14:22:32.906175Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ProductName', 'EngineVersion', 'AppVersion', 'AvSigVersion', 'RtpStateBitfield', 'IsSxsPassiveMode', 'AVProductsInstalled', 'AVProductsEnabled', 'HasTpm', 'CountryIdentifier', 'OrganizationIdentifier', 'GeoNameIdentifier', 'LocaleEnglishNameIdentifier', 'Platform', 'Processor', 'OsVer', 'OsBuild', 'OsSuite', 'OsPlatformSubRelease', 'SkuEdition', 'IsProtected', 'SMode', 'IeVerIdentifier', 'SmartScreen', 'Firewall', 'Census_MDC2FormFactor', 'Census_ProcessorCoreCount', 'Census_ProcessorManufacturerIdentifier', 'Census_PrimaryDiskTypeName', 'Census_HasOpticalDiskDrive', 'Census_ChassisTypeName', 'Census_PowerPlatformRoleName', 'Census_InternalBatteryType', 'Census_OSVersion', 'Census_OSArchitecture', 'Census_OSBranch', 'Census_OSBuildNumber', 'Census_OSBuildRevision', 'Census_OSEdition', 'Census_OSSkuName', 'Census_OSInstallTypeName', 'Census_OSInstallLanguageIdentifier', 'Census_OSUILocaleIdentifier', 'Census_OSWUAutoUpdateOptionsName', 'Census_GenuineStateName', 'Census_ActivationChannel', 'Census_IsFlightingInternal', 'Census_IsFlightsDisabled', 'Census_FlightRing', 'Census_ThresholdOptIn', 'Census_IsSecureBootEnabled', 'Census_IsWIMBootEnabled', 'Census_IsTouchEnabled', 'Census_IsPenCapable', 'Census_IsAlwaysOnAlwaysConnectedCapable', 'Wdft_IsGamer', 'Wdft_RegionIdentifier']\n"
     ]
    }
   ],
   "source": [
    "true_numerical_columns = [\n",
    "    'Census_PrimaryDiskTotalCapacity', 'Census_SystemVolumeTotalCapacity',\n",
    "    'Census_TotalPhysicalRAM', 'Census_InternalBatteryNumberOfCharges'\n",
    "]\n",
    "\n",
    "categorical_columns_high_car = [\n",
    "    'Census_FirmwareVersionIdentifier', 'Census_OEMModelIdentifier',\n",
    "    'AVProductStatesIdentifier', 'Census_FirmwareManufacturerIdentifier',\n",
    "    'Census_InternalPrimaryDiagonalDisplaySizeInInches',\n",
    "    'Census_InternalPrimaryDisplayResolutionHorizontal',\n",
    "    'Census_InternalPrimaryDisplayResolutionVertical',\n",
    "    'Census_OEMNameIdentifier', 'Census_ProcessorModelIdentifier',\n",
    "    'CityIdentifier', 'DefaultBrowsersIdentifier', 'OsBuildLab'\n",
    "]\n",
    "\n",
    "categorical_columns = [\n",
    "    c for c in df_all.columns\n",
    "    if c not in (['HasDetections', 'Date'] + true_numerical_columns +\n",
    "                 categorical_columns_high_car)\n",
    "]\n",
    "print(categorical_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 编码 -- Label Encoding "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为将使用的模型是[LightGBM](https://lightgbm.readthedocs.io/en/latest/)，所以我们需要对分类变量做编码。\n",
    "\n",
    "这里用的方法是[Label Encoding](http://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.factorize.html)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:22:34.853958Z",
     "start_time": "2019-04-29T14:22:33.160340Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 69/69 [00:01<00:00, 40.89it/s]\n"
     ]
    }
   ],
   "source": [
    "def factor_data(df, col):\n",
    "    df_labeled, _ = df[col].factorize(sort=True)\n",
    "    # MAKE SMALLEST LABEL 1, RESERVE 0\n",
    "    df_labeled += 1\n",
    "    # MAKE NAN LARGEST LABEL\n",
    "    df_labeled = np.where(df_labeled==0, df_labeled.max()+1, df_labeled)\n",
    "    df[col] = df_labeled\n",
    "\n",
    "for col in tqdm(categorical_columns + categorical_columns_high_car):\n",
    "    factor_data(df_all, col) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构造测试集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "像刚才提到的，因为没有使用测试集的数据，所以我们需要从训练集中拆分出一个数据集，作为我们的测试集，用于评价我们评估模型的方式是否有效。\n",
    "\n",
    "因为训练集和测试集是根据时间划分的，所以我们从训练集拆分的测试集，同样也根据时间划分。\n",
    "\n",
    "这是为了尽量模拟真实的测试集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:22:35.392771Z",
     "start_time": "2019-04-29T14:22:35.104945Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 将样本根据时间排序\n",
    "df_all = df_all.sort_values('Date').reset_index(drop=True) \n",
    "df_all.drop(['Date'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:22:35.600550Z",
     "start_time": "2019-04-29T14:22:35.594674Z"
    }
   },
   "outputs": [],
   "source": [
    "# 将前80%的样本作为训练集，后20%的样本作为测试集\n",
    "df_test = df_all.iloc[int(0.8*len(df_all)):, ]\n",
    "df_train = df_all.iloc[:int(0.8*len(df_all)), ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 对抗验证（Adversarial Validatiion）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:23:30.443891Z",
     "start_time": "2019-04-29T14:22:36.309834Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tcv_agg's auc: 0.981538 + 0.000780946\n",
      "[2]\tcv_agg's auc: 0.991886 + 0.0011813\n",
      "[3]\tcv_agg's auc: 0.997781 + 0.00022218\n",
      "[4]\tcv_agg's auc: 0.998255 + 0.000357413\n",
      "[5]\tcv_agg's auc: 0.999215 + 0.000110628\n",
      "[6]\tcv_agg's auc: 0.999382 + 4.43679e-05\n",
      "[7]\tcv_agg's auc: 0.999477 + 3.37784e-05\n",
      "[8]\tcv_agg's auc: 0.999488 + 4.45144e-05\n",
      "[9]\tcv_agg's auc: 0.999513 + 4.41991e-05\n",
      "[10]\tcv_agg's auc: 0.999587 + 4.13675e-05\n",
      "[11]\tcv_agg's auc: 0.999624 + 5.00616e-05\n",
      "[12]\tcv_agg's auc: 0.999663 + 3.64739e-05\n",
      "[13]\tcv_agg's auc: 0.999683 + 3.87281e-05\n",
      "[14]\tcv_agg's auc: 0.999708 + 1.98319e-05\n",
      "[15]\tcv_agg's auc: 0.999731 + 7.15555e-06\n",
      "[16]\tcv_agg's auc: 0.999756 + 2.11395e-05\n",
      "[17]\tcv_agg's auc: 0.999771 + 1.06916e-05\n",
      "[18]\tcv_agg's auc: 0.999799 + 2.30115e-05\n",
      "[19]\tcv_agg's auc: 0.999816 + 1.94767e-05\n",
      "[20]\tcv_agg's auc: 0.999836 + 2.90241e-05\n",
      "[21]\tcv_agg's auc: 0.999859 + 3.2347e-05\n",
      "[22]\tcv_agg's auc: 0.999867 + 2.67817e-05\n",
      "[23]\tcv_agg's auc: 0.999874 + 3.19173e-05\n",
      "[24]\tcv_agg's auc: 0.999875 + 2.13535e-05\n",
      "[25]\tcv_agg's auc: 0.999883 + 3.20498e-05\n",
      "[26]\tcv_agg's auc: 0.999891 + 3.29026e-05\n",
      "[27]\tcv_agg's auc: 0.999909 + 3.67415e-05\n",
      "[28]\tcv_agg's auc: 0.999914 + 3.4628e-05\n",
      "[29]\tcv_agg's auc: 0.999915 + 3.9002e-05\n",
      "[30]\tcv_agg's auc: 0.999928 + 2.73712e-05\n",
      "[31]\tcv_agg's auc: 0.999931 + 3.72592e-05\n",
      "[32]\tcv_agg's auc: 0.999942 + 2.61211e-05\n",
      "[33]\tcv_agg's auc: 0.999941 + 2.8799e-05\n",
      "[34]\tcv_agg's auc: 0.999942 + 2.24311e-05\n",
      "[35]\tcv_agg's auc: 0.999948 + 2.51907e-05\n",
      "[36]\tcv_agg's auc: 0.99995 + 2.21959e-05\n",
      "[37]\tcv_agg's auc: 0.999951 + 2.44169e-05\n",
      "[38]\tcv_agg's auc: 0.999953 + 1.99818e-05\n",
      "[39]\tcv_agg's auc: 0.999957 + 1.96095e-05\n",
      "[40]\tcv_agg's auc: 0.999959 + 1.99055e-05\n",
      "[41]\tcv_agg's auc: 0.999962 + 1.55492e-05\n",
      "[42]\tcv_agg's auc: 0.999964 + 1.73165e-05\n",
      "[43]\tcv_agg's auc: 0.999966 + 1.76397e-05\n",
      "[44]\tcv_agg's auc: 0.999968 + 1.60664e-05\n",
      "[45]\tcv_agg's auc: 0.99997 + 1.54404e-05\n",
      "[46]\tcv_agg's auc: 0.999971 + 1.36191e-05\n",
      "[47]\tcv_agg's auc: 0.999973 + 1.26826e-05\n",
      "[48]\tcv_agg's auc: 0.999974 + 1.24906e-05\n",
      "[49]\tcv_agg's auc: 0.999975 + 1.07114e-05\n",
      "[50]\tcv_agg's auc: 0.999976 + 1.0567e-05\n",
      "[51]\tcv_agg's auc: 0.999978 + 9.97035e-06\n",
      "[52]\tcv_agg's auc: 0.99998 + 1.0196e-05\n",
      "[53]\tcv_agg's auc: 0.999982 + 6.21599e-06\n",
      "[54]\tcv_agg's auc: 0.999982 + 6.21175e-06\n",
      "[55]\tcv_agg's auc: 0.999984 + 5.21395e-06\n",
      "[56]\tcv_agg's auc: 0.999984 + 4.55237e-06\n",
      "[57]\tcv_agg's auc: 0.999985 + 4.43549e-06\n",
      "[58]\tcv_agg's auc: 0.999985 + 4.14502e-06\n",
      "[59]\tcv_agg's auc: 0.999986 + 4.24647e-06\n",
      "[60]\tcv_agg's auc: 0.999986 + 4.46867e-06\n",
      "[61]\tcv_agg's auc: 0.999986 + 4.5085e-06\n",
      "[62]\tcv_agg's auc: 0.999987 + 4.55437e-06\n",
      "[63]\tcv_agg's auc: 0.999987 + 3.84372e-06\n",
      "[64]\tcv_agg's auc: 0.999988 + 3.69414e-06\n",
      "[65]\tcv_agg's auc: 0.999988 + 3.8886e-06\n",
      "[66]\tcv_agg's auc: 0.999989 + 4.26284e-06\n",
      "[67]\tcv_agg's auc: 0.999989 + 4.45168e-06\n",
      "[68]\tcv_agg's auc: 0.999989 + 4.7677e-06\n",
      "[69]\tcv_agg's auc: 0.999989 + 4.75466e-06\n",
      "[70]\tcv_agg's auc: 0.999989 + 4.8537e-06\n",
      "[71]\tcv_agg's auc: 0.999989 + 4.91339e-06\n",
      "[72]\tcv_agg's auc: 0.999989 + 4.83621e-06\n",
      "[73]\tcv_agg's auc: 0.999989 + 4.97557e-06\n",
      "[74]\tcv_agg's auc: 0.999989 + 5.0407e-06\n",
      "[75]\tcv_agg's auc: 0.999989 + 5.02808e-06\n",
      "[76]\tcv_agg's auc: 0.99999 + 5.05205e-06\n",
      "[77]\tcv_agg's auc: 0.999989 + 5.46558e-06\n",
      "[78]\tcv_agg's auc: 0.999989 + 5.38276e-06\n",
      "[79]\tcv_agg's auc: 0.999989 + 5.44731e-06\n",
      "[80]\tcv_agg's auc: 0.999989 + 5.37485e-06\n",
      "[81]\tcv_agg's auc: 0.999989 + 5.51855e-06\n",
      "[82]\tcv_agg's auc: 0.999989 + 5.59476e-06\n",
      "[83]\tcv_agg's auc: 0.999989 + 5.62625e-06\n",
      "[84]\tcv_agg's auc: 0.999989 + 6.00121e-06\n",
      "[85]\tcv_agg's auc: 0.999989 + 5.58734e-06\n",
      "[86]\tcv_agg's auc: 0.999989 + 5.70356e-06\n",
      "[87]\tcv_agg's auc: 0.999989 + 5.71859e-06\n",
      "[88]\tcv_agg's auc: 0.999989 + 5.49096e-06\n",
      "[89]\tcv_agg's auc: 0.999989 + 5.42575e-06\n",
      "[90]\tcv_agg's auc: 0.999989 + 5.60569e-06\n",
      "[91]\tcv_agg's auc: 0.999989 + 5.60696e-06\n",
      "[92]\tcv_agg's auc: 0.999989 + 5.57031e-06\n",
      "[93]\tcv_agg's auc: 0.999989 + 5.81221e-06\n",
      "[94]\tcv_agg's auc: 0.999989 + 5.78214e-06\n",
      "[95]\tcv_agg's auc: 0.99999 + 5.50068e-06\n",
      "[96]\tcv_agg's auc: 0.999989 + 5.78288e-06\n",
      "[97]\tcv_agg's auc: 0.999989 + 6.07988e-06\n",
      "[98]\tcv_agg's auc: 0.999989 + 6.0739e-06\n",
      "[99]\tcv_agg's auc: 0.999989 + 6.10934e-06\n",
      "[100]\tcv_agg's auc: 0.999989 + 6.19086e-06\n",
      "[101]\tcv_agg's auc: 0.999989 + 6.28362e-06\n",
      "[102]\tcv_agg's auc: 0.999989 + 6.21196e-06\n",
      "[103]\tcv_agg's auc: 0.999989 + 6.05199e-06\n",
      "[104]\tcv_agg's auc: 0.999989 + 6.31287e-06\n",
      "[105]\tcv_agg's auc: 0.999989 + 6.36231e-06\n",
      "[106]\tcv_agg's auc: 0.999989 + 6.10983e-06\n",
      "[107]\tcv_agg's auc: 0.999989 + 5.93104e-06\n",
      "[108]\tcv_agg's auc: 0.999989 + 5.91942e-06\n",
      "[109]\tcv_agg's auc: 0.999989 + 5.6606e-06\n",
      "[110]\tcv_agg's auc: 0.999989 + 5.68739e-06\n",
      "[111]\tcv_agg's auc: 0.999989 + 5.56753e-06\n",
      "[112]\tcv_agg's auc: 0.999989 + 5.57126e-06\n",
      "[113]\tcv_agg's auc: 0.999989 + 6.47731e-06\n",
      "[114]\tcv_agg's auc: 0.999989 + 6.58044e-06\n",
      "[115]\tcv_agg's auc: 0.999989 + 6.34622e-06\n",
      "[116]\tcv_agg's auc: 0.999989 + 5.55571e-06\n",
      "[117]\tcv_agg's auc: 0.99999 + 5.19579e-06\n",
      "[118]\tcv_agg's auc: 0.99999 + 5.25817e-06\n",
      "[119]\tcv_agg's auc: 0.99999 + 5.19185e-06\n",
      "[120]\tcv_agg's auc: 0.99999 + 5.16857e-06\n",
      "[121]\tcv_agg's auc: 0.99999 + 5.09913e-06\n",
      "[122]\tcv_agg's auc: 0.99999 + 5.06656e-06\n",
      "[123]\tcv_agg's auc: 0.99999 + 5.23252e-06\n",
      "[124]\tcv_agg's auc: 0.99999 + 5.28021e-06\n",
      "[125]\tcv_agg's auc: 0.99999 + 5.20594e-06\n",
      "[126]\tcv_agg's auc: 0.999989 + 6.2853e-06\n",
      "[127]\tcv_agg's auc: 0.999989 + 6.38375e-06\n",
      "[128]\tcv_agg's auc: 0.999989 + 6.38462e-06\n",
      "[129]\tcv_agg's auc: 0.999989 + 6.44221e-06\n",
      "[130]\tcv_agg's auc: 0.999988 + 7.02864e-06\n",
      "[131]\tcv_agg's auc: 0.999989 + 5.75995e-06\n",
      "[132]\tcv_agg's auc: 0.999989 + 5.33974e-06\n",
      "[133]\tcv_agg's auc: 0.999989 + 5.19658e-06\n",
      "[134]\tcv_agg's auc: 0.999989 + 4.90274e-06\n",
      "[135]\tcv_agg's auc: 0.999989 + 4.87911e-06\n",
      "[136]\tcv_agg's auc: 0.999989 + 4.89881e-06\n",
      "[137]\tcv_agg's auc: 0.999989 + 4.86725e-06\n",
      "[138]\tcv_agg's auc: 0.999989 + 5.04453e-06\n",
      "[139]\tcv_agg's auc: 0.999989 + 5.1598e-06\n",
      "[140]\tcv_agg's auc: 0.999989 + 5.13723e-06\n",
      "[141]\tcv_agg's auc: 0.999989 + 5.2381e-06\n",
      "[142]\tcv_agg's auc: 0.999989 + 5.26486e-06\n",
      "[143]\tcv_agg's auc: 0.999989 + 5.58018e-06\n",
      "[144]\tcv_agg's auc: 0.999988 + 5.84606e-06\n",
      "[145]\tcv_agg's auc: 0.999988 + 5.84041e-06\n",
      "[146]\tcv_agg's auc: 0.999988 + 5.93474e-06\n",
      "[147]\tcv_agg's auc: 0.999988 + 5.89872e-06\n",
      "[148]\tcv_agg's auc: 0.999988 + 6.04723e-06\n",
      "[149]\tcv_agg's auc: 0.999988 + 6.00463e-06\n",
      "[150]\tcv_agg's auc: 0.999988 + 5.7565e-06\n",
      "[151]\tcv_agg's auc: 0.999988 + 5.69513e-06\n",
      "[152]\tcv_agg's auc: 0.999988 + 5.99798e-06\n",
      "[153]\tcv_agg's auc: 0.999987 + 6.50533e-06\n",
      "[154]\tcv_agg's auc: 0.999987 + 6.63042e-06\n",
      "[155]\tcv_agg's auc: 0.999987 + 6.51145e-06\n",
      "[156]\tcv_agg's auc: 0.999987 + 6.44731e-06\n",
      "[157]\tcv_agg's auc: 0.999987 + 6.41089e-06\n",
      "[158]\tcv_agg's auc: 0.999987 + 6.85672e-06\n",
      "[159]\tcv_agg's auc: 0.999987 + 6.9946e-06\n",
      "[160]\tcv_agg's auc: 0.999987 + 7.26798e-06\n",
      "[161]\tcv_agg's auc: 0.999987 + 7.26389e-06\n",
      "[162]\tcv_agg's auc: 0.999987 + 7.38817e-06\n",
      "[163]\tcv_agg's auc: 0.999987 + 7.60873e-06\n",
      "[164]\tcv_agg's auc: 0.999987 + 7.53663e-06\n",
      "[165]\tcv_agg's auc: 0.999986 + 7.46187e-06\n",
      "[166]\tcv_agg's auc: 0.999986 + 7.58908e-06\n",
      "[167]\tcv_agg's auc: 0.999986 + 8.07343e-06\n",
      "[168]\tcv_agg's auc: 0.999986 + 8.0905e-06\n",
      "[169]\tcv_agg's auc: 0.999986 + 8.14856e-06\n",
      "[170]\tcv_agg's auc: 0.999986 + 8.13082e-06\n",
      "[171]\tcv_agg's auc: 0.999986 + 8.23315e-06\n",
      "[172]\tcv_agg's auc: 0.999986 + 7.472e-06\n",
      "[173]\tcv_agg's auc: 0.999986 + 7.39294e-06\n",
      "[174]\tcv_agg's auc: 0.999986 + 7.06596e-06\n",
      "[175]\tcv_agg's auc: 0.999986 + 6.79754e-06\n",
      "[176]\tcv_agg's auc: 0.999986 + 6.73456e-06\n",
      "[177]\tcv_agg's auc: 0.999986 + 6.95924e-06\n",
      "[178]\tcv_agg's auc: 0.999986 + 7.18976e-06\n",
      "[179]\tcv_agg's auc: 0.999985 + 7.08264e-06\n",
      "[180]\tcv_agg's auc: 0.999985 + 7.20777e-06\n",
      "[181]\tcv_agg's auc: 0.999985 + 7.29514e-06\n",
      "[182]\tcv_agg's auc: 0.999985 + 7.21728e-06\n",
      "[183]\tcv_agg's auc: 0.999985 + 7.19856e-06\n",
      "[184]\tcv_agg's auc: 0.999985 + 7.44719e-06\n",
      "[185]\tcv_agg's auc: 0.999985 + 7.36883e-06\n",
      "[186]\tcv_agg's auc: 0.999985 + 7.32237e-06\n",
      "[187]\tcv_agg's auc: 0.999985 + 7.2697e-06\n",
      "[188]\tcv_agg's auc: 0.999985 + 7.18456e-06\n",
      "[189]\tcv_agg's auc: 0.999985 + 7.02324e-06\n",
      "[190]\tcv_agg's auc: 0.999985 + 7.0995e-06\n",
      "[191]\tcv_agg's auc: 0.999985 + 7.08608e-06\n",
      "[192]\tcv_agg's auc: 0.999985 + 7.03873e-06\n",
      "[193]\tcv_agg's auc: 0.999985 + 7.12676e-06\n",
      "[194]\tcv_agg's auc: 0.999985 + 7.07631e-06\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[195]\tcv_agg's auc: 0.999985 + 6.99561e-06\n",
      "[196]\tcv_agg's auc: 0.999985 + 7.02045e-06\n",
      "[197]\tcv_agg's auc: 0.999985 + 7.35668e-06\n",
      "[198]\tcv_agg's auc: 0.999984 + 7.90237e-06\n",
      "[199]\tcv_agg's auc: 0.999984 + 7.65815e-06\n",
      "[200]\tcv_agg's auc: 0.999984 + 7.71088e-06\n",
      "[201]\tcv_agg's auc: 0.999984 + 7.65929e-06\n",
      "[202]\tcv_agg's auc: 0.999984 + 7.44411e-06\n",
      "[203]\tcv_agg's auc: 0.999984 + 7.26212e-06\n",
      "[204]\tcv_agg's auc: 0.999985 + 6.71573e-06\n",
      "[205]\tcv_agg's auc: 0.999985 + 6.73701e-06\n",
      "[206]\tcv_agg's auc: 0.999985 + 6.7113e-06\n",
      "[207]\tcv_agg's auc: 0.999985 + 6.58684e-06\n",
      "[208]\tcv_agg's auc: 0.999985 + 6.76216e-06\n",
      "[209]\tcv_agg's auc: 0.999985 + 6.84919e-06\n",
      "[210]\tcv_agg's auc: 0.999984 + 6.64034e-06\n",
      "[211]\tcv_agg's auc: 0.999984 + 6.57349e-06\n",
      "[212]\tcv_agg's auc: 0.999984 + 6.43365e-06\n",
      "[213]\tcv_agg's auc: 0.999985 + 5.99357e-06\n",
      "[214]\tcv_agg's auc: 0.999985 + 6.07347e-06\n",
      "[215]\tcv_agg's auc: 0.999985 + 6.1671e-06\n",
      "[216]\tcv_agg's auc: 0.999985 + 6.09936e-06\n",
      "[217]\tcv_agg's auc: 0.999985 + 6.03495e-06\n",
      "[218]\tcv_agg's auc: 0.999985 + 6.15025e-06\n",
      "[219]\tcv_agg's auc: 0.999985 + 6.06694e-06\n",
      "[220]\tcv_agg's auc: 0.999984 + 6.0539e-06\n",
      "[221]\tcv_agg's auc: 0.999984 + 6.00197e-06\n",
      "[222]\tcv_agg's auc: 0.999984 + 6.16698e-06\n",
      "[223]\tcv_agg's auc: 0.999984 + 6.01828e-06\n",
      "[224]\tcv_agg's auc: 0.999984 + 5.95932e-06\n",
      "[225]\tcv_agg's auc: 0.999984 + 5.86445e-06\n",
      "[226]\tcv_agg's auc: 0.999984 + 5.93437e-06\n",
      "[227]\tcv_agg's auc: 0.999984 + 5.74969e-06\n",
      "[228]\tcv_agg's auc: 0.999984 + 5.90268e-06\n",
      "[229]\tcv_agg's auc: 0.999984 + 5.86288e-06\n",
      "[230]\tcv_agg's auc: 0.999984 + 5.96892e-06\n",
      "[231]\tcv_agg's auc: 0.999984 + 6.28413e-06\n",
      "[232]\tcv_agg's auc: 0.999984 + 6.33159e-06\n",
      "[233]\tcv_agg's auc: 0.999984 + 6.34636e-06\n",
      "[234]\tcv_agg's auc: 0.999983 + 6.53198e-06\n",
      "[235]\tcv_agg's auc: 0.999983 + 6.83487e-06\n",
      "[236]\tcv_agg's auc: 0.999983 + 6.89899e-06\n",
      "[237]\tcv_agg's auc: 0.999983 + 6.65531e-06\n",
      "[238]\tcv_agg's auc: 0.999983 + 6.54576e-06\n",
      "[239]\tcv_agg's auc: 0.999983 + 6.5402e-06\n",
      "[240]\tcv_agg's auc: 0.999983 + 6.38336e-06\n",
      "[241]\tcv_agg's auc: 0.999983 + 6.53471e-06\n",
      "[242]\tcv_agg's auc: 0.999983 + 6.45193e-06\n",
      "[243]\tcv_agg's auc: 0.999983 + 6.29234e-06\n",
      "[244]\tcv_agg's auc: 0.999983 + 6.24745e-06\n",
      "[245]\tcv_agg's auc: 0.999983 + 6.28599e-06\n",
      "[246]\tcv_agg's auc: 0.999982 + 6.48079e-06\n",
      "[247]\tcv_agg's auc: 0.999982 + 6.54037e-06\n",
      "[248]\tcv_agg's auc: 0.999982 + 6.99155e-06\n",
      "[249]\tcv_agg's auc: 0.999982 + 6.78032e-06\n",
      "[250]\tcv_agg's auc: 0.999982 + 6.87417e-06\n",
      "[251]\tcv_agg's auc: 0.999982 + 6.82163e-06\n",
      "[252]\tcv_agg's auc: 0.999982 + 6.76351e-06\n",
      "[253]\tcv_agg's auc: 0.999982 + 6.82122e-06\n",
      "[254]\tcv_agg's auc: 0.999982 + 6.95264e-06\n",
      "[255]\tcv_agg's auc: 0.999982 + 6.92523e-06\n",
      "[256]\tcv_agg's auc: 0.999982 + 6.86982e-06\n",
      "[257]\tcv_agg's auc: 0.999981 + 7.18047e-06\n",
      "[258]\tcv_agg's auc: 0.999981 + 7.27232e-06\n",
      "[259]\tcv_agg's auc: 0.999981 + 7.2314e-06\n",
      "[260]\tcv_agg's auc: 0.999981 + 7.29005e-06\n",
      "[261]\tcv_agg's auc: 0.999981 + 7.25903e-06\n",
      "[262]\tcv_agg's auc: 0.999981 + 7.20086e-06\n",
      "[263]\tcv_agg's auc: 0.999981 + 7.30105e-06\n",
      "[264]\tcv_agg's auc: 0.999981 + 7.22296e-06\n",
      "[265]\tcv_agg's auc: 0.999981 + 7.25268e-06\n",
      "[266]\tcv_agg's auc: 0.999981 + 7.13516e-06\n",
      "[267]\tcv_agg's auc: 0.999981 + 7.16063e-06\n",
      "[268]\tcv_agg's auc: 0.999981 + 7.14735e-06\n",
      "[269]\tcv_agg's auc: 0.999981 + 7.21517e-06\n",
      "[270]\tcv_agg's auc: 0.999981 + 7.37327e-06\n",
      "[271]\tcv_agg's auc: 0.99998 + 7.5145e-06\n",
      "[272]\tcv_agg's auc: 0.99998 + 7.23621e-06\n",
      "[273]\tcv_agg's auc: 0.99998 + 7.30897e-06\n",
      "[274]\tcv_agg's auc: 0.99998 + 7.16732e-06\n",
      "[275]\tcv_agg's auc: 0.99998 + 7.13814e-06\n",
      "[276]\tcv_agg's auc: 0.99998 + 7.18034e-06\n",
      "[277]\tcv_agg's auc: 0.99998 + 7.19658e-06\n",
      "[278]\tcv_agg's auc: 0.99998 + 7.1888e-06\n",
      "[279]\tcv_agg's auc: 0.99998 + 7.20014e-06\n",
      "[280]\tcv_agg's auc: 0.99998 + 7.23141e-06\n",
      "[281]\tcv_agg's auc: 0.99998 + 7.27401e-06\n",
      "[282]\tcv_agg's auc: 0.99998 + 7.40622e-06\n",
      "[283]\tcv_agg's auc: 0.999979 + 7.47748e-06\n",
      "[284]\tcv_agg's auc: 0.999979 + 7.51758e-06\n",
      "[285]\tcv_agg's auc: 0.999979 + 7.5804e-06\n",
      "[286]\tcv_agg's auc: 0.999979 + 7.65509e-06\n",
      "[287]\tcv_agg's auc: 0.999979 + 7.84858e-06\n",
      "[288]\tcv_agg's auc: 0.999979 + 7.92811e-06\n",
      "[289]\tcv_agg's auc: 0.999979 + 7.98349e-06\n",
      "[290]\tcv_agg's auc: 0.999979 + 7.92739e-06\n",
      "[291]\tcv_agg's auc: 0.999979 + 8.13448e-06\n",
      "[292]\tcv_agg's auc: 0.999979 + 8.06636e-06\n",
      "[293]\tcv_agg's auc: 0.999979 + 7.89025e-06\n",
      "[294]\tcv_agg's auc: 0.999979 + 7.97416e-06\n",
      "[295]\tcv_agg's auc: 0.999978 + 7.86417e-06\n",
      "[296]\tcv_agg's auc: 0.999978 + 8.08502e-06\n",
      "[297]\tcv_agg's auc: 0.999978 + 7.79679e-06\n",
      "[298]\tcv_agg's auc: 0.999978 + 7.70481e-06\n",
      "[299]\tcv_agg's auc: 0.999978 + 7.76873e-06\n",
      "[300]\tcv_agg's auc: 0.999978 + 7.75833e-06\n",
      "[301]\tcv_agg's auc: 0.999978 + 7.65831e-06\n",
      "[302]\tcv_agg's auc: 0.999978 + 7.77774e-06\n",
      "[303]\tcv_agg's auc: 0.999978 + 7.62811e-06\n",
      "[304]\tcv_agg's auc: 0.999978 + 7.68167e-06\n",
      "[305]\tcv_agg's auc: 0.999978 + 7.81949e-06\n",
      "[306]\tcv_agg's auc: 0.999978 + 7.9807e-06\n",
      "[307]\tcv_agg's auc: 0.999978 + 8.00497e-06\n",
      "[308]\tcv_agg's auc: 0.999978 + 7.97678e-06\n",
      "[309]\tcv_agg's auc: 0.999978 + 7.70708e-06\n",
      "[310]\tcv_agg's auc: 0.999978 + 7.69278e-06\n",
      "[311]\tcv_agg's auc: 0.999977 + 7.64955e-06\n",
      "[312]\tcv_agg's auc: 0.999977 + 7.63303e-06\n",
      "[313]\tcv_agg's auc: 0.999977 + 7.81239e-06\n",
      "[314]\tcv_agg's auc: 0.999978 + 7.79484e-06\n",
      "[315]\tcv_agg's auc: 0.999978 + 7.73821e-06\n",
      "[316]\tcv_agg's auc: 0.999978 + 7.84518e-06\n",
      "[317]\tcv_agg's auc: 0.999978 + 7.82312e-06\n",
      "[318]\tcv_agg's auc: 0.999978 + 7.79197e-06\n",
      "[319]\tcv_agg's auc: 0.999978 + 7.80681e-06\n",
      "[320]\tcv_agg's auc: 0.999977 + 7.75131e-06\n",
      "交叉验证中最优的AUC为 0.99999，对应的标准差为0.00001.\n",
      "模型最优的迭代次数为120.\n"
     ]
    }
   ],
   "source": [
    "# 定义新的Y\n",
    "df_train['Is_Test'] = 0\n",
    "df_test['Is_Test'] = 1\n",
    "\n",
    "# 将 Train 和 Test 合成一个数据集。HasDetections是数据本来的Y，所以剔除。\n",
    "df_adv = pd.concat([df_train, df_test])\n",
    "\n",
    "adv_data = lgb.Dataset(\n",
    "    data=df_adv.drop('Is_Test', axis=1), label=df_adv.loc[:, 'Is_Test'])\n",
    "\n",
    "# 定义模型参数\n",
    "params = {\n",
    "    'boosting_type': 'gbdt',\n",
    "    'colsample_bytree': 1,\n",
    "    'learning_rate': 0.1,\n",
    "    'max_depth': 5,\n",
    "    'min_child_samples': 100,\n",
    "    'min_child_weight': 1,\n",
    "    'min_split_gain': 0.0,\n",
    "    'num_leaves': 20,\n",
    "    'objective': 'binary',\n",
    "    'random_state': 50,\n",
    "    'subsample': 1.0,\n",
    "    'subsample_freq': 0,\n",
    "    'metric': 'auc',\n",
    "    'num_threads': 8\n",
    "}\n",
    "\n",
    "# 交叉验证\n",
    "adv_cv_results = lgb.cv(\n",
    "    params,\n",
    "    adv_data,\n",
    "    num_boost_round=10000,\n",
    "    nfold=5,\n",
    "    categorical_feature=categorical_columns,\n",
    "    early_stopping_rounds=200,\n",
    "    verbose_eval=True,\n",
    "    seed=42)\n",
    "\n",
    "print('交叉验证中最优的AUC为 {:.5f}，对应的标准差为{:.5f}.'.format(\n",
    "    adv_cv_results['auc-mean'][-1], adv_cv_results['auc-stdv'][-1]))\n",
    "\n",
    "print('模型最优的迭代次数为{}.'.format(len(adv_cv_results['auc-mean'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过对抗验证，我们发现模型的AUC达到了0.99。说明本次比赛的训练集和测试集的样本分布存在较大的差异。\n",
    "\n",
    "然后，我们使用训练好的模型，对所有的样本进行预测，得到各个样本属于测试集的概率。这个之后会用到。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:23:36.121156Z",
     "start_time": "2019-04-29T14:23:30.905411Z"
    }
   },
   "outputs": [],
   "source": [
    "params['n_estimators'] = len(adv_cv_results['auc-mean'])\n",
    "\n",
    "model_adv = lgb.LGBMClassifier(**params)\n",
    "model_adv.fit(df_adv.drop('Is_Test', axis=1), df_adv.loc[:, 'Is_Test'])\n",
    "\n",
    "preds_adv = model_adv.predict_proba(df_adv.drop('Is_Test', axis=1))[:, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-25T08:46:01.150039Z",
     "start_time": "2019-04-25T08:46:01.146049Z"
    }
   },
   "source": [
    "## 交叉验证（Cross Validation）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们知道了训练集和测试集的分布存在很大的差异。那么接下来，我们采用交叉验证的方法，来评估模型的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:23:36.617052Z",
     "start_time": "2019-04-29T14:23:36.605748Z"
    }
   },
   "outputs": [],
   "source": [
    "def run_cv(df_train, sample_weight=None):\n",
    "    if sample_weight is not None:\n",
    "        train_set = lgb.Dataset(\n",
    "            df_train.drop('HasDetections', axis=1),\n",
    "            label=df_train.loc[:, 'HasDetections'], weight=sample_weight)\n",
    "\n",
    "    else:\n",
    "        train_set = lgb.Dataset(\n",
    "            df_train.drop('HasDetections', axis=1),\n",
    "            label=df_train.loc[:, 'HasDetections'])\n",
    "\n",
    "    # Perform cross validation with early stopping\n",
    "    params.pop('n_estimators', None)\n",
    "    \n",
    "    N_FOLDS = 5\n",
    "    cv_results = lgb.cv(\n",
    "        params,\n",
    "        train_set,\n",
    "        num_boost_round=10000,\n",
    "        nfold=N_FOLDS,\n",
    "        categorical_feature=categorical_columns,\n",
    "        early_stopping_rounds=200,\n",
    "        verbose_eval=True,\n",
    "        seed=42)\n",
    "\n",
    "    print('交叉验证中最优的AUC为 {:.5f}，对应的标准差为{:.5f}.'.format(\n",
    "        cv_results['auc-mean'][-1], cv_results['auc-stdv'][-1]))\n",
    "\n",
    "    print('模型最优的迭代次数为{}.'.format(len(cv_results['auc-mean'])))\n",
    "\n",
    "    params['n_estimators'] = len(cv_results['auc-mean'])\n",
    "\n",
    "    model_cv = lgb.LGBMClassifier(**params)\n",
    "    model_cv.fit(df_train.drop('HasDetections', axis=1),\n",
    "                 df_train.loc[:, 'HasDetections'])\n",
    "\n",
    "    # AUC\n",
    "    preds_test_cv = model_cv.predict_proba(\n",
    "        df_test.drop('HasDetections', axis=1))[:, 1]\n",
    "    auc_test_cv = roc_auc_score(df_test.loc[:, 'HasDetections'], preds_test_cv)\n",
    "    print('模型在测试集上的效果是{:.5f}。'.format(\n",
    "        auc_test_cv))\n",
    "\n",
    "    return model_cv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:24:23.375363Z",
     "start_time": "2019-04-29T14:23:36.954170Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tcv_agg's auc: 0.673642 + 0.00265052\n",
      "[2]\tcv_agg's auc: 0.678893 + 0.00448619\n",
      "[3]\tcv_agg's auc: 0.681506 + 0.00347297\n",
      "[4]\tcv_agg's auc: 0.682945 + 0.00385881\n",
      "[5]\tcv_agg's auc: 0.684684 + 0.0041531\n",
      "[6]\tcv_agg's auc: 0.686289 + 0.00343128\n",
      "[7]\tcv_agg's auc: 0.687549 + 0.00316743\n",
      "[8]\tcv_agg's auc: 0.688257 + 0.0034225\n",
      "[9]\tcv_agg's auc: 0.689064 + 0.00333125\n",
      "[10]\tcv_agg's auc: 0.690081 + 0.00332507\n",
      "[11]\tcv_agg's auc: 0.690898 + 0.00355769\n",
      "[12]\tcv_agg's auc: 0.691641 + 0.00343357\n",
      "[13]\tcv_agg's auc: 0.692108 + 0.00359948\n",
      "[14]\tcv_agg's auc: 0.692707 + 0.00384981\n",
      "[15]\tcv_agg's auc: 0.693462 + 0.00350958\n",
      "[16]\tcv_agg's auc: 0.693609 + 0.00341991\n",
      "[17]\tcv_agg's auc: 0.69383 + 0.00328001\n",
      "[18]\tcv_agg's auc: 0.694183 + 0.00329412\n",
      "[19]\tcv_agg's auc: 0.694634 + 0.00340595\n",
      "[20]\tcv_agg's auc: 0.695012 + 0.00318532\n",
      "[21]\tcv_agg's auc: 0.69514 + 0.00324965\n",
      "[22]\tcv_agg's auc: 0.695565 + 0.00343536\n",
      "[23]\tcv_agg's auc: 0.696031 + 0.00331183\n",
      "[24]\tcv_agg's auc: 0.696424 + 0.00332546\n",
      "[25]\tcv_agg's auc: 0.696618 + 0.00332648\n",
      "[26]\tcv_agg's auc: 0.697007 + 0.00344113\n",
      "[27]\tcv_agg's auc: 0.697209 + 0.00354729\n",
      "[28]\tcv_agg's auc: 0.69753 + 0.00339477\n",
      "[29]\tcv_agg's auc: 0.697669 + 0.00349056\n",
      "[30]\tcv_agg's auc: 0.697982 + 0.00354894\n",
      "[31]\tcv_agg's auc: 0.698157 + 0.00368691\n",
      "[32]\tcv_agg's auc: 0.698417 + 0.00357652\n",
      "[33]\tcv_agg's auc: 0.698542 + 0.00356946\n",
      "[34]\tcv_agg's auc: 0.698682 + 0.00346246\n",
      "[35]\tcv_agg's auc: 0.698788 + 0.00346071\n",
      "[36]\tcv_agg's auc: 0.698829 + 0.00349142\n",
      "[37]\tcv_agg's auc: 0.698899 + 0.00352678\n",
      "[38]\tcv_agg's auc: 0.699016 + 0.00352905\n",
      "[39]\tcv_agg's auc: 0.699233 + 0.00359655\n",
      "[40]\tcv_agg's auc: 0.699332 + 0.00368476\n",
      "[41]\tcv_agg's auc: 0.699447 + 0.00366172\n",
      "[42]\tcv_agg's auc: 0.699669 + 0.00350924\n",
      "[43]\tcv_agg's auc: 0.699811 + 0.0035476\n",
      "[44]\tcv_agg's auc: 0.699935 + 0.00350795\n",
      "[45]\tcv_agg's auc: 0.700074 + 0.00359138\n",
      "[46]\tcv_agg's auc: 0.700116 + 0.00349727\n",
      "[47]\tcv_agg's auc: 0.700237 + 0.00348833\n",
      "[48]\tcv_agg's auc: 0.700189 + 0.00352063\n",
      "[49]\tcv_agg's auc: 0.700248 + 0.00353862\n",
      "[50]\tcv_agg's auc: 0.700359 + 0.00347187\n",
      "[51]\tcv_agg's auc: 0.700308 + 0.00340073\n",
      "[52]\tcv_agg's auc: 0.700455 + 0.00341802\n",
      "[53]\tcv_agg's auc: 0.700528 + 0.00347858\n",
      "[54]\tcv_agg's auc: 0.700496 + 0.0035206\n",
      "[55]\tcv_agg's auc: 0.700625 + 0.0034804\n",
      "[56]\tcv_agg's auc: 0.700623 + 0.00350384\n",
      "[57]\tcv_agg's auc: 0.700556 + 0.003555\n",
      "[58]\tcv_agg's auc: 0.700757 + 0.00348334\n",
      "[59]\tcv_agg's auc: 0.70087 + 0.0036437\n",
      "[60]\tcv_agg's auc: 0.700844 + 0.00360128\n",
      "[61]\tcv_agg's auc: 0.700907 + 0.00350601\n",
      "[62]\tcv_agg's auc: 0.700944 + 0.00358943\n",
      "[63]\tcv_agg's auc: 0.700936 + 0.00359792\n",
      "[64]\tcv_agg's auc: 0.700895 + 0.00361589\n",
      "[65]\tcv_agg's auc: 0.70107 + 0.00351954\n",
      "[66]\tcv_agg's auc: 0.701037 + 0.0035337\n",
      "[67]\tcv_agg's auc: 0.701161 + 0.00352953\n",
      "[68]\tcv_agg's auc: 0.701156 + 0.00354453\n",
      "[69]\tcv_agg's auc: 0.70117 + 0.00360473\n",
      "[70]\tcv_agg's auc: 0.701303 + 0.00356851\n",
      "[71]\tcv_agg's auc: 0.701282 + 0.00351559\n",
      "[72]\tcv_agg's auc: 0.701268 + 0.00350478\n",
      "[73]\tcv_agg's auc: 0.701277 + 0.00351383\n",
      "[74]\tcv_agg's auc: 0.701225 + 0.00353991\n",
      "[75]\tcv_agg's auc: 0.701169 + 0.00356183\n",
      "[76]\tcv_agg's auc: 0.701243 + 0.00348941\n",
      "[77]\tcv_agg's auc: 0.701183 + 0.00352386\n",
      "[78]\tcv_agg's auc: 0.701221 + 0.00351123\n",
      "[79]\tcv_agg's auc: 0.701222 + 0.00349972\n",
      "[80]\tcv_agg's auc: 0.701289 + 0.00341932\n",
      "[81]\tcv_agg's auc: 0.701352 + 0.00342555\n",
      "[82]\tcv_agg's auc: 0.70136 + 0.003446\n",
      "[83]\tcv_agg's auc: 0.701444 + 0.0034886\n",
      "[84]\tcv_agg's auc: 0.701396 + 0.00342717\n",
      "[85]\tcv_agg's auc: 0.701337 + 0.00339179\n",
      "[86]\tcv_agg's auc: 0.701374 + 0.00336005\n",
      "[87]\tcv_agg's auc: 0.701321 + 0.00333481\n",
      "[88]\tcv_agg's auc: 0.701301 + 0.00338374\n",
      "[89]\tcv_agg's auc: 0.701298 + 0.0033618\n",
      "[90]\tcv_agg's auc: 0.701278 + 0.00336204\n",
      "[91]\tcv_agg's auc: 0.701265 + 0.00335858\n",
      "[92]\tcv_agg's auc: 0.701315 + 0.00335322\n",
      "[93]\tcv_agg's auc: 0.701317 + 0.0033131\n",
      "[94]\tcv_agg's auc: 0.701312 + 0.00335443\n",
      "[95]\tcv_agg's auc: 0.7013 + 0.0033588\n",
      "[96]\tcv_agg's auc: 0.701277 + 0.00340577\n",
      "[97]\tcv_agg's auc: 0.701336 + 0.00342654\n",
      "[98]\tcv_agg's auc: 0.701353 + 0.0033602\n",
      "[99]\tcv_agg's auc: 0.701342 + 0.00334934\n",
      "[100]\tcv_agg's auc: 0.701333 + 0.00327826\n",
      "[101]\tcv_agg's auc: 0.701373 + 0.00315796\n",
      "[102]\tcv_agg's auc: 0.701338 + 0.00313672\n",
      "[103]\tcv_agg's auc: 0.701358 + 0.00313454\n",
      "[104]\tcv_agg's auc: 0.701334 + 0.00310083\n",
      "[105]\tcv_agg's auc: 0.701349 + 0.00311119\n",
      "[106]\tcv_agg's auc: 0.701344 + 0.00307943\n",
      "[107]\tcv_agg's auc: 0.701361 + 0.00311846\n",
      "[108]\tcv_agg's auc: 0.70133 + 0.00307583\n",
      "[109]\tcv_agg's auc: 0.701325 + 0.0030723\n",
      "[110]\tcv_agg's auc: 0.701298 + 0.00308351\n",
      "[111]\tcv_agg's auc: 0.701273 + 0.00312572\n",
      "[112]\tcv_agg's auc: 0.701263 + 0.0031083\n",
      "[113]\tcv_agg's auc: 0.70126 + 0.00320346\n",
      "[114]\tcv_agg's auc: 0.70113 + 0.00317217\n",
      "[115]\tcv_agg's auc: 0.701108 + 0.00314055\n",
      "[116]\tcv_agg's auc: 0.70113 + 0.00313596\n",
      "[117]\tcv_agg's auc: 0.701161 + 0.0031374\n",
      "[118]\tcv_agg's auc: 0.701144 + 0.00312682\n",
      "[119]\tcv_agg's auc: 0.70114 + 0.00308709\n",
      "[120]\tcv_agg's auc: 0.701105 + 0.00308077\n",
      "[121]\tcv_agg's auc: 0.701082 + 0.00306223\n",
      "[122]\tcv_agg's auc: 0.701048 + 0.00310531\n",
      "[123]\tcv_agg's auc: 0.701017 + 0.00305225\n",
      "[124]\tcv_agg's auc: 0.700998 + 0.00297471\n",
      "[125]\tcv_agg's auc: 0.701031 + 0.00295602\n",
      "[126]\tcv_agg's auc: 0.701038 + 0.00299307\n",
      "[127]\tcv_agg's auc: 0.701018 + 0.00295206\n",
      "[128]\tcv_agg's auc: 0.700954 + 0.00298988\n",
      "[129]\tcv_agg's auc: 0.700985 + 0.00297788\n",
      "[130]\tcv_agg's auc: 0.700951 + 0.00299711\n",
      "[131]\tcv_agg's auc: 0.700989 + 0.00305483\n",
      "[132]\tcv_agg's auc: 0.700965 + 0.00302485\n",
      "[133]\tcv_agg's auc: 0.700975 + 0.00299782\n",
      "[134]\tcv_agg's auc: 0.700974 + 0.00302281\n",
      "[135]\tcv_agg's auc: 0.700928 + 0.0030342\n",
      "[136]\tcv_agg's auc: 0.700925 + 0.00304591\n",
      "[137]\tcv_agg's auc: 0.700917 + 0.00297527\n",
      "[138]\tcv_agg's auc: 0.700903 + 0.00302708\n",
      "[139]\tcv_agg's auc: 0.700964 + 0.00306983\n",
      "[140]\tcv_agg's auc: 0.700986 + 0.00306126\n",
      "[141]\tcv_agg's auc: 0.700945 + 0.00305566\n",
      "[142]\tcv_agg's auc: 0.700942 + 0.00303926\n",
      "[143]\tcv_agg's auc: 0.700945 + 0.00304662\n",
      "[144]\tcv_agg's auc: 0.700904 + 0.00303724\n",
      "[145]\tcv_agg's auc: 0.700892 + 0.0030165\n",
      "[146]\tcv_agg's auc: 0.70087 + 0.00305019\n",
      "[147]\tcv_agg's auc: 0.700864 + 0.0030495\n",
      "[148]\tcv_agg's auc: 0.700788 + 0.00307361\n",
      "[149]\tcv_agg's auc: 0.700771 + 0.00311525\n",
      "[150]\tcv_agg's auc: 0.7007 + 0.00308477\n",
      "[151]\tcv_agg's auc: 0.700727 + 0.00305555\n",
      "[152]\tcv_agg's auc: 0.70072 + 0.00306656\n",
      "[153]\tcv_agg's auc: 0.700675 + 0.00305017\n",
      "[154]\tcv_agg's auc: 0.700674 + 0.00302731\n",
      "[155]\tcv_agg's auc: 0.700646 + 0.00307325\n",
      "[156]\tcv_agg's auc: 0.700626 + 0.00306582\n",
      "[157]\tcv_agg's auc: 0.700594 + 0.0030525\n",
      "[158]\tcv_agg's auc: 0.70061 + 0.00302189\n",
      "[159]\tcv_agg's auc: 0.700597 + 0.00301431\n",
      "[160]\tcv_agg's auc: 0.700626 + 0.00303968\n",
      "[161]\tcv_agg's auc: 0.700623 + 0.00310973\n",
      "[162]\tcv_agg's auc: 0.700627 + 0.00316075\n",
      "[163]\tcv_agg's auc: 0.700593 + 0.00313325\n",
      "[164]\tcv_agg's auc: 0.700546 + 0.0031815\n",
      "[165]\tcv_agg's auc: 0.700564 + 0.00317237\n",
      "[166]\tcv_agg's auc: 0.700562 + 0.00321054\n",
      "[167]\tcv_agg's auc: 0.700588 + 0.00322513\n",
      "[168]\tcv_agg's auc: 0.700594 + 0.00324684\n",
      "[169]\tcv_agg's auc: 0.700591 + 0.00319782\n",
      "[170]\tcv_agg's auc: 0.700611 + 0.00321006\n",
      "[171]\tcv_agg's auc: 0.700598 + 0.00320638\n",
      "[172]\tcv_agg's auc: 0.700629 + 0.00323349\n",
      "[173]\tcv_agg's auc: 0.700611 + 0.00323962\n",
      "[174]\tcv_agg's auc: 0.700559 + 0.00321616\n",
      "[175]\tcv_agg's auc: 0.70055 + 0.0032438\n",
      "[176]\tcv_agg's auc: 0.700511 + 0.00324205\n",
      "[177]\tcv_agg's auc: 0.700443 + 0.00328424\n",
      "[178]\tcv_agg's auc: 0.700423 + 0.00330401\n",
      "[179]\tcv_agg's auc: 0.700403 + 0.00334071\n",
      "[180]\tcv_agg's auc: 0.700331 + 0.00335042\n",
      "[181]\tcv_agg's auc: 0.700305 + 0.00332576\n",
      "[182]\tcv_agg's auc: 0.700304 + 0.00332936\n",
      "[183]\tcv_agg's auc: 0.700276 + 0.00336139\n",
      "[184]\tcv_agg's auc: 0.700278 + 0.00338539\n",
      "[185]\tcv_agg's auc: 0.700273 + 0.00345021\n",
      "[186]\tcv_agg's auc: 0.700256 + 0.00345276\n",
      "[187]\tcv_agg's auc: 0.700259 + 0.00341567\n",
      "[188]\tcv_agg's auc: 0.7002 + 0.00341557\n",
      "[189]\tcv_agg's auc: 0.700175 + 0.0034026\n",
      "[190]\tcv_agg's auc: 0.70025 + 0.00338391\n",
      "[191]\tcv_agg's auc: 0.700189 + 0.00340941\n",
      "[192]\tcv_agg's auc: 0.700185 + 0.00336489\n",
      "[193]\tcv_agg's auc: 0.700175 + 0.00337788\n",
      "[194]\tcv_agg's auc: 0.700116 + 0.0033633\n",
      "[195]\tcv_agg's auc: 0.700113 + 0.00335926\n",
      "[196]\tcv_agg's auc: 0.70013 + 0.00334293\n",
      "[197]\tcv_agg's auc: 0.700017 + 0.00336249\n",
      "[198]\tcv_agg's auc: 0.699991 + 0.00336951\n",
      "[199]\tcv_agg's auc: 0.699957 + 0.00334875\n",
      "[200]\tcv_agg's auc: 0.699931 + 0.00338514\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[201]\tcv_agg's auc: 0.699909 + 0.00332299\n",
      "[202]\tcv_agg's auc: 0.699903 + 0.003358\n",
      "[203]\tcv_agg's auc: 0.699889 + 0.00332874\n",
      "[204]\tcv_agg's auc: 0.699886 + 0.00328193\n",
      "[205]\tcv_agg's auc: 0.699943 + 0.00320245\n",
      "[206]\tcv_agg's auc: 0.699879 + 0.00322782\n",
      "[207]\tcv_agg's auc: 0.699792 + 0.0032461\n",
      "[208]\tcv_agg's auc: 0.699732 + 0.00325164\n",
      "[209]\tcv_agg's auc: 0.69971 + 0.00319165\n",
      "[210]\tcv_agg's auc: 0.699629 + 0.00319212\n",
      "[211]\tcv_agg's auc: 0.699643 + 0.00320773\n",
      "[212]\tcv_agg's auc: 0.699651 + 0.00324131\n",
      "[213]\tcv_agg's auc: 0.699618 + 0.00330046\n",
      "[214]\tcv_agg's auc: 0.699601 + 0.00327513\n",
      "[215]\tcv_agg's auc: 0.699595 + 0.00326767\n",
      "[216]\tcv_agg's auc: 0.69956 + 0.00324504\n",
      "[217]\tcv_agg's auc: 0.699553 + 0.0032811\n",
      "[218]\tcv_agg's auc: 0.699539 + 0.00330446\n",
      "[219]\tcv_agg's auc: 0.699516 + 0.00330204\n",
      "[220]\tcv_agg's auc: 0.699502 + 0.00329667\n",
      "[221]\tcv_agg's auc: 0.699485 + 0.00328499\n",
      "[222]\tcv_agg's auc: 0.699456 + 0.00328292\n",
      "[223]\tcv_agg's auc: 0.69946 + 0.00325112\n",
      "[224]\tcv_agg's auc: 0.699471 + 0.00325479\n",
      "[225]\tcv_agg's auc: 0.699463 + 0.00327326\n",
      "[226]\tcv_agg's auc: 0.699395 + 0.00331988\n",
      "[227]\tcv_agg's auc: 0.699334 + 0.00330879\n",
      "[228]\tcv_agg's auc: 0.699289 + 0.00331747\n",
      "[229]\tcv_agg's auc: 0.69926 + 0.00337595\n",
      "[230]\tcv_agg's auc: 0.699193 + 0.00340362\n",
      "[231]\tcv_agg's auc: 0.699178 + 0.0033784\n",
      "[232]\tcv_agg's auc: 0.699108 + 0.00339935\n",
      "[233]\tcv_agg's auc: 0.699053 + 0.0034054\n",
      "[234]\tcv_agg's auc: 0.699005 + 0.00339277\n",
      "[235]\tcv_agg's auc: 0.699033 + 0.00337479\n",
      "[236]\tcv_agg's auc: 0.699018 + 0.00338721\n",
      "[237]\tcv_agg's auc: 0.699028 + 0.00340033\n",
      "[238]\tcv_agg's auc: 0.698996 + 0.00338473\n",
      "[239]\tcv_agg's auc: 0.699049 + 0.00341858\n",
      "[240]\tcv_agg's auc: 0.699041 + 0.00344354\n",
      "[241]\tcv_agg's auc: 0.699018 + 0.00345283\n",
      "[242]\tcv_agg's auc: 0.699005 + 0.00344224\n",
      "[243]\tcv_agg's auc: 0.698971 + 0.00343403\n",
      "[244]\tcv_agg's auc: 0.698889 + 0.00338428\n",
      "[245]\tcv_agg's auc: 0.698806 + 0.00340342\n",
      "[246]\tcv_agg's auc: 0.698796 + 0.0033782\n",
      "[247]\tcv_agg's auc: 0.698774 + 0.00339589\n",
      "[248]\tcv_agg's auc: 0.698726 + 0.00339153\n",
      "[249]\tcv_agg's auc: 0.698727 + 0.00339353\n",
      "[250]\tcv_agg's auc: 0.698671 + 0.00340723\n",
      "[251]\tcv_agg's auc: 0.698662 + 0.00342493\n",
      "[252]\tcv_agg's auc: 0.698616 + 0.00342843\n",
      "[253]\tcv_agg's auc: 0.698588 + 0.00345922\n",
      "[254]\tcv_agg's auc: 0.698618 + 0.00340707\n",
      "[255]\tcv_agg's auc: 0.698631 + 0.00338383\n",
      "[256]\tcv_agg's auc: 0.698589 + 0.0033882\n",
      "[257]\tcv_agg's auc: 0.698604 + 0.00339151\n",
      "[258]\tcv_agg's auc: 0.698586 + 0.00342305\n",
      "[259]\tcv_agg's auc: 0.698598 + 0.00345163\n",
      "[260]\tcv_agg's auc: 0.698658 + 0.00350502\n",
      "[261]\tcv_agg's auc: 0.698618 + 0.00347696\n",
      "[262]\tcv_agg's auc: 0.6986 + 0.00348389\n",
      "[263]\tcv_agg's auc: 0.698579 + 0.00344392\n",
      "[264]\tcv_agg's auc: 0.698526 + 0.00340998\n",
      "[265]\tcv_agg's auc: 0.69851 + 0.00338996\n",
      "[266]\tcv_agg's auc: 0.698501 + 0.0034598\n",
      "[267]\tcv_agg's auc: 0.698485 + 0.0034776\n",
      "[268]\tcv_agg's auc: 0.698472 + 0.00348485\n",
      "[269]\tcv_agg's auc: 0.698484 + 0.00345757\n",
      "[270]\tcv_agg's auc: 0.698455 + 0.00344144\n",
      "[271]\tcv_agg's auc: 0.698478 + 0.00343007\n",
      "[272]\tcv_agg's auc: 0.698429 + 0.0034687\n",
      "[273]\tcv_agg's auc: 0.698374 + 0.00348816\n",
      "[274]\tcv_agg's auc: 0.698337 + 0.00346743\n",
      "[275]\tcv_agg's auc: 0.698319 + 0.00344545\n",
      "[276]\tcv_agg's auc: 0.698274 + 0.00344278\n",
      "[277]\tcv_agg's auc: 0.698256 + 0.00341593\n",
      "[278]\tcv_agg's auc: 0.698239 + 0.0034148\n",
      "[279]\tcv_agg's auc: 0.698214 + 0.00339488\n",
      "[280]\tcv_agg's auc: 0.698186 + 0.00341459\n",
      "[281]\tcv_agg's auc: 0.698182 + 0.00338633\n",
      "[282]\tcv_agg's auc: 0.69816 + 0.00342484\n",
      "[283]\tcv_agg's auc: 0.698171 + 0.00339367\n",
      "交叉验证中最优的AUC为 0.70144，对应的标准差为0.00349.\n",
      "模型最优的迭代次数为83.\n",
      "模型在测试集上的效果是0.66980。\n"
     ]
    }
   ],
   "source": [
    "model_cv = run_cv(df_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用交叉验证的方式来评估模型效果：\n",
    "\n",
    "* 交叉验证AUC：0.70144\n",
    "* 测试集上AUC：0.66980\n",
    "* 差值：0.03\n",
    "\n",
    "交叉验证和测试集上的AUC差值较大，说明交叉验证的方式不太能准确评估模型在测试集上的效果。 \n",
    "\n",
    "我们再来来试一下其他方法，对比看看。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 在变量分布变化的情况下，除了交叉验证，还有哪些更优的方法？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 人工划分验证集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:24:23.865992Z",
     "start_time": "2019-04-29T14:24:23.855557Z"
    }
   },
   "outputs": [],
   "source": [
    "def run_lgb(df_train, df_validation):\n",
    "    dtrain = lgb.Dataset(\n",
    "        data=df_train.drop('HasDetections', axis=1),\n",
    "        label=df_train.loc[:, 'HasDetections'],\n",
    "        free_raw_data=False,\n",
    "        silent=True)\n",
    "\n",
    "    dvalid = lgb.Dataset(\n",
    "        data=df_validation.drop('HasDetections', axis=1),\n",
    "        label=df_validation.loc[:, 'HasDetections'],\n",
    "        free_raw_data=False,\n",
    "        silent=True)\n",
    "\n",
    "    params.pop('n_estimators', None)\n",
    "\n",
    "    clf = lgb.train(\n",
    "        params=params,\n",
    "        train_set=dtrain,\n",
    "        num_boost_round=10000,\n",
    "        valid_sets=[dtrain, dvalid],\n",
    "        early_stopping_rounds=200,\n",
    "        verbose_eval=True,\n",
    "        categorical_feature=categorical_columns)\n",
    "\n",
    "    params['n_estimators'] = clf.num_trees()\n",
    "\n",
    "    model = lgb.LGBMClassifier(**params)\n",
    "    model.fit(\n",
    "        df_train.drop('HasDetections', axis=1),\n",
    "        df_train.loc[:, 'HasDetections'])\n",
    "\n",
    "    # AUC\n",
    "    preds_test = model.predict_proba(\n",
    "        df_test.drop('HasDetections', axis=1))[:, 1]\n",
    "    auc_test = roc_auc_score(df_test.loc[:, 'HasDetections'], preds_test)\n",
    "    print('模型在测试集上的效果是{:.5f}。'.format(\n",
    "        roc_auc_score(df_test.loc[:, 'HasDetections'], preds_test)))\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:24:24.378877Z",
     "start_time": "2019-04-29T14:24:24.372931Z"
    }
   },
   "outputs": [],
   "source": [
    "# 之前已经用Date进行了排序，所以提取出后20%的样本作为验证集。\n",
    "df_validation_1 = df_train.iloc[int(0.8 * len(df_train)):, ]\n",
    "df_train_1 = df_train.iloc[:int(0.8 * len(df_train)), ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:24:52.071739Z",
     "start_time": "2019-04-29T14:24:24.865799Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\ttraining's auc: 0.688654\tvalid_1's auc: 0.642369\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[2]\ttraining's auc: 0.69429\tvalid_1's auc: 0.649962\n",
      "[3]\ttraining's auc: 0.696418\tvalid_1's auc: 0.651748\n",
      "[4]\ttraining's auc: 0.698366\tvalid_1's auc: 0.653697\n",
      "[5]\ttraining's auc: 0.699429\tvalid_1's auc: 0.654109\n",
      "[6]\ttraining's auc: 0.70092\tvalid_1's auc: 0.654634\n",
      "[7]\ttraining's auc: 0.704122\tvalid_1's auc: 0.655645\n",
      "[8]\ttraining's auc: 0.706059\tvalid_1's auc: 0.657076\n",
      "[9]\ttraining's auc: 0.706789\tvalid_1's auc: 0.657017\n",
      "[10]\ttraining's auc: 0.708772\tvalid_1's auc: 0.659064\n",
      "[11]\ttraining's auc: 0.709966\tvalid_1's auc: 0.659128\n",
      "[12]\ttraining's auc: 0.710595\tvalid_1's auc: 0.65896\n",
      "[13]\ttraining's auc: 0.712962\tvalid_1's auc: 0.660827\n",
      "[14]\ttraining's auc: 0.714985\tvalid_1's auc: 0.66104\n",
      "[15]\ttraining's auc: 0.716574\tvalid_1's auc: 0.661911\n",
      "[16]\ttraining's auc: 0.717889\tvalid_1's auc: 0.662541\n",
      "[17]\ttraining's auc: 0.718961\tvalid_1's auc: 0.663312\n",
      "[18]\ttraining's auc: 0.719463\tvalid_1's auc: 0.663325\n",
      "[19]\ttraining's auc: 0.720344\tvalid_1's auc: 0.663683\n",
      "[20]\ttraining's auc: 0.721599\tvalid_1's auc: 0.664064\n",
      "[21]\ttraining's auc: 0.722693\tvalid_1's auc: 0.664566\n",
      "[22]\ttraining's auc: 0.724333\tvalid_1's auc: 0.664592\n",
      "[23]\ttraining's auc: 0.726076\tvalid_1's auc: 0.664567\n",
      "[24]\ttraining's auc: 0.726912\tvalid_1's auc: 0.664831\n",
      "[25]\ttraining's auc: 0.727809\tvalid_1's auc: 0.665285\n",
      "[26]\ttraining's auc: 0.728751\tvalid_1's auc: 0.665552\n",
      "[27]\ttraining's auc: 0.729567\tvalid_1's auc: 0.665861\n",
      "[28]\ttraining's auc: 0.730693\tvalid_1's auc: 0.666068\n",
      "[29]\ttraining's auc: 0.731295\tvalid_1's auc: 0.66627\n",
      "[30]\ttraining's auc: 0.732656\tvalid_1's auc: 0.666682\n",
      "[31]\ttraining's auc: 0.734244\tvalid_1's auc: 0.667238\n",
      "[32]\ttraining's auc: 0.734617\tvalid_1's auc: 0.667438\n",
      "[33]\ttraining's auc: 0.735755\tvalid_1's auc: 0.667634\n",
      "[34]\ttraining's auc: 0.736473\tvalid_1's auc: 0.667857\n",
      "[35]\ttraining's auc: 0.737041\tvalid_1's auc: 0.668193\n",
      "[36]\ttraining's auc: 0.737953\tvalid_1's auc: 0.668424\n",
      "[37]\ttraining's auc: 0.738672\tvalid_1's auc: 0.668604\n",
      "[38]\ttraining's auc: 0.739306\tvalid_1's auc: 0.668619\n",
      "[39]\ttraining's auc: 0.740149\tvalid_1's auc: 0.668806\n",
      "[40]\ttraining's auc: 0.741027\tvalid_1's auc: 0.668736\n",
      "[41]\ttraining's auc: 0.7418\tvalid_1's auc: 0.669289\n",
      "[42]\ttraining's auc: 0.742798\tvalid_1's auc: 0.669182\n",
      "[43]\ttraining's auc: 0.743543\tvalid_1's auc: 0.669401\n",
      "[44]\ttraining's auc: 0.744133\tvalid_1's auc: 0.66942\n",
      "[45]\ttraining's auc: 0.744889\tvalid_1's auc: 0.669675\n",
      "[46]\ttraining's auc: 0.745673\tvalid_1's auc: 0.669923\n",
      "[47]\ttraining's auc: 0.746163\tvalid_1's auc: 0.669996\n",
      "[48]\ttraining's auc: 0.746685\tvalid_1's auc: 0.670011\n",
      "[49]\ttraining's auc: 0.747231\tvalid_1's auc: 0.669929\n",
      "[50]\ttraining's auc: 0.747828\tvalid_1's auc: 0.669891\n",
      "[51]\ttraining's auc: 0.748548\tvalid_1's auc: 0.670154\n",
      "[52]\ttraining's auc: 0.748941\tvalid_1's auc: 0.670129\n",
      "[53]\ttraining's auc: 0.749524\tvalid_1's auc: 0.67022\n",
      "[54]\ttraining's auc: 0.75009\tvalid_1's auc: 0.670376\n",
      "[55]\ttraining's auc: 0.7505\tvalid_1's auc: 0.670401\n",
      "[56]\ttraining's auc: 0.75081\tvalid_1's auc: 0.670476\n",
      "[57]\ttraining's auc: 0.751278\tvalid_1's auc: 0.670438\n",
      "[58]\ttraining's auc: 0.751738\tvalid_1's auc: 0.67058\n",
      "[59]\ttraining's auc: 0.752876\tvalid_1's auc: 0.670933\n",
      "[60]\ttraining's auc: 0.753329\tvalid_1's auc: 0.671118\n",
      "[61]\ttraining's auc: 0.754042\tvalid_1's auc: 0.671232\n",
      "[62]\ttraining's auc: 0.754507\tvalid_1's auc: 0.67097\n",
      "[63]\ttraining's auc: 0.755058\tvalid_1's auc: 0.671048\n",
      "[64]\ttraining's auc: 0.755727\tvalid_1's auc: 0.67111\n",
      "[65]\ttraining's auc: 0.755917\tvalid_1's auc: 0.671214\n",
      "[66]\ttraining's auc: 0.756408\tvalid_1's auc: 0.671567\n",
      "[67]\ttraining's auc: 0.756805\tvalid_1's auc: 0.671921\n",
      "[68]\ttraining's auc: 0.757773\tvalid_1's auc: 0.67199\n",
      "[69]\ttraining's auc: 0.758474\tvalid_1's auc: 0.672291\n",
      "[70]\ttraining's auc: 0.759006\tvalid_1's auc: 0.672362\n",
      "[71]\ttraining's auc: 0.759274\tvalid_1's auc: 0.672374\n",
      "[72]\ttraining's auc: 0.759438\tvalid_1's auc: 0.672354\n",
      "[73]\ttraining's auc: 0.759981\tvalid_1's auc: 0.67238\n",
      "[74]\ttraining's auc: 0.760568\tvalid_1's auc: 0.672358\n",
      "[75]\ttraining's auc: 0.761004\tvalid_1's auc: 0.672336\n",
      "[76]\ttraining's auc: 0.761584\tvalid_1's auc: 0.672316\n",
      "[77]\ttraining's auc: 0.761843\tvalid_1's auc: 0.672249\n",
      "[78]\ttraining's auc: 0.762365\tvalid_1's auc: 0.672334\n",
      "[79]\ttraining's auc: 0.762798\tvalid_1's auc: 0.672063\n",
      "[80]\ttraining's auc: 0.76338\tvalid_1's auc: 0.672229\n",
      "[81]\ttraining's auc: 0.763979\tvalid_1's auc: 0.672253\n",
      "[82]\ttraining's auc: 0.764841\tvalid_1's auc: 0.672218\n",
      "[83]\ttraining's auc: 0.765111\tvalid_1's auc: 0.672163\n",
      "[84]\ttraining's auc: 0.765522\tvalid_1's auc: 0.672151\n",
      "[85]\ttraining's auc: 0.766144\tvalid_1's auc: 0.672274\n",
      "[86]\ttraining's auc: 0.766643\tvalid_1's auc: 0.672333\n",
      "[87]\ttraining's auc: 0.766981\tvalid_1's auc: 0.672419\n",
      "[88]\ttraining's auc: 0.767287\tvalid_1's auc: 0.672742\n",
      "[89]\ttraining's auc: 0.76751\tvalid_1's auc: 0.672715\n",
      "[90]\ttraining's auc: 0.768206\tvalid_1's auc: 0.672682\n",
      "[91]\ttraining's auc: 0.768888\tvalid_1's auc: 0.672751\n",
      "[92]\ttraining's auc: 0.769377\tvalid_1's auc: 0.672744\n",
      "[93]\ttraining's auc: 0.769932\tvalid_1's auc: 0.67283\n",
      "[94]\ttraining's auc: 0.770681\tvalid_1's auc: 0.672915\n",
      "[95]\ttraining's auc: 0.771331\tvalid_1's auc: 0.673069\n",
      "[96]\ttraining's auc: 0.771935\tvalid_1's auc: 0.673173\n",
      "[97]\ttraining's auc: 0.772238\tvalid_1's auc: 0.673153\n",
      "[98]\ttraining's auc: 0.772439\tvalid_1's auc: 0.673206\n",
      "[99]\ttraining's auc: 0.772888\tvalid_1's auc: 0.673318\n",
      "[100]\ttraining's auc: 0.773327\tvalid_1's auc: 0.673488\n",
      "[101]\ttraining's auc: 0.773751\tvalid_1's auc: 0.673426\n",
      "[102]\ttraining's auc: 0.773971\tvalid_1's auc: 0.673473\n",
      "[103]\ttraining's auc: 0.774134\tvalid_1's auc: 0.67344\n",
      "[104]\ttraining's auc: 0.774524\tvalid_1's auc: 0.67339\n",
      "[105]\ttraining's auc: 0.774771\tvalid_1's auc: 0.673443\n",
      "[106]\ttraining's auc: 0.775062\tvalid_1's auc: 0.673437\n",
      "[107]\ttraining's auc: 0.775578\tvalid_1's auc: 0.673569\n",
      "[108]\ttraining's auc: 0.776031\tvalid_1's auc: 0.673551\n",
      "[109]\ttraining's auc: 0.776539\tvalid_1's auc: 0.67365\n",
      "[110]\ttraining's auc: 0.777196\tvalid_1's auc: 0.673689\n",
      "[111]\ttraining's auc: 0.777382\tvalid_1's auc: 0.673831\n",
      "[112]\ttraining's auc: 0.777807\tvalid_1's auc: 0.673789\n",
      "[113]\ttraining's auc: 0.778069\tvalid_1's auc: 0.673776\n",
      "[114]\ttraining's auc: 0.778449\tvalid_1's auc: 0.673827\n",
      "[115]\ttraining's auc: 0.77896\tvalid_1's auc: 0.673841\n",
      "[116]\ttraining's auc: 0.779176\tvalid_1's auc: 0.673792\n",
      "[117]\ttraining's auc: 0.779302\tvalid_1's auc: 0.673889\n",
      "[118]\ttraining's auc: 0.779998\tvalid_1's auc: 0.673906\n",
      "[119]\ttraining's auc: 0.780686\tvalid_1's auc: 0.674032\n",
      "[120]\ttraining's auc: 0.781048\tvalid_1's auc: 0.674109\n",
      "[121]\ttraining's auc: 0.781401\tvalid_1's auc: 0.674198\n",
      "[122]\ttraining's auc: 0.781686\tvalid_1's auc: 0.674133\n",
      "[123]\ttraining's auc: 0.782201\tvalid_1's auc: 0.674294\n",
      "[124]\ttraining's auc: 0.782744\tvalid_1's auc: 0.674398\n",
      "[125]\ttraining's auc: 0.782977\tvalid_1's auc: 0.674409\n",
      "[126]\ttraining's auc: 0.783571\tvalid_1's auc: 0.674427\n",
      "[127]\ttraining's auc: 0.783779\tvalid_1's auc: 0.67437\n",
      "[128]\ttraining's auc: 0.784331\tvalid_1's auc: 0.674475\n",
      "[129]\ttraining's auc: 0.784855\tvalid_1's auc: 0.674389\n",
      "[130]\ttraining's auc: 0.785285\tvalid_1's auc: 0.674353\n",
      "[131]\ttraining's auc: 0.78538\tvalid_1's auc: 0.674372\n",
      "[132]\ttraining's auc: 0.785786\tvalid_1's auc: 0.674438\n",
      "[133]\ttraining's auc: 0.785979\tvalid_1's auc: 0.674508\n",
      "[134]\ttraining's auc: 0.78644\tvalid_1's auc: 0.674573\n",
      "[135]\ttraining's auc: 0.786732\tvalid_1's auc: 0.6747\n",
      "[136]\ttraining's auc: 0.787158\tvalid_1's auc: 0.674808\n",
      "[137]\ttraining's auc: 0.787596\tvalid_1's auc: 0.674956\n",
      "[138]\ttraining's auc: 0.787942\tvalid_1's auc: 0.674977\n",
      "[139]\ttraining's auc: 0.78815\tvalid_1's auc: 0.674924\n",
      "[140]\ttraining's auc: 0.788499\tvalid_1's auc: 0.674942\n",
      "[141]\ttraining's auc: 0.788862\tvalid_1's auc: 0.675068\n",
      "[142]\ttraining's auc: 0.789064\tvalid_1's auc: 0.675044\n",
      "[143]\ttraining's auc: 0.789238\tvalid_1's auc: 0.675025\n",
      "[144]\ttraining's auc: 0.789706\tvalid_1's auc: 0.674957\n",
      "[145]\ttraining's auc: 0.789917\tvalid_1's auc: 0.674989\n",
      "[146]\ttraining's auc: 0.790375\tvalid_1's auc: 0.675019\n",
      "[147]\ttraining's auc: 0.790722\tvalid_1's auc: 0.675062\n",
      "[148]\ttraining's auc: 0.790781\tvalid_1's auc: 0.675094\n",
      "[149]\ttraining's auc: 0.790979\tvalid_1's auc: 0.675084\n",
      "[150]\ttraining's auc: 0.791175\tvalid_1's auc: 0.67507\n",
      "[151]\ttraining's auc: 0.791465\tvalid_1's auc: 0.675013\n",
      "[152]\ttraining's auc: 0.791613\tvalid_1's auc: 0.675035\n",
      "[153]\ttraining's auc: 0.79213\tvalid_1's auc: 0.675027\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[154]\ttraining's auc: 0.792496\tvalid_1's auc: 0.675089\n",
      "[155]\ttraining's auc: 0.792854\tvalid_1's auc: 0.675041\n",
      "[156]\ttraining's auc: 0.793164\tvalid_1's auc: 0.67512\n",
      "[157]\ttraining's auc: 0.793717\tvalid_1's auc: 0.67515\n",
      "[158]\ttraining's auc: 0.794199\tvalid_1's auc: 0.675206\n",
      "[159]\ttraining's auc: 0.794418\tvalid_1's auc: 0.675215\n",
      "[160]\ttraining's auc: 0.794797\tvalid_1's auc: 0.675297\n",
      "[161]\ttraining's auc: 0.795373\tvalid_1's auc: 0.675284\n",
      "[162]\ttraining's auc: 0.795846\tvalid_1's auc: 0.67528\n",
      "[163]\ttraining's auc: 0.796273\tvalid_1's auc: 0.675307\n",
      "[164]\ttraining's auc: 0.796541\tvalid_1's auc: 0.675257\n",
      "[165]\ttraining's auc: 0.796964\tvalid_1's auc: 0.675325\n",
      "[166]\ttraining's auc: 0.797298\tvalid_1's auc: 0.675346\n",
      "[167]\ttraining's auc: 0.797701\tvalid_1's auc: 0.675336\n",
      "[168]\ttraining's auc: 0.798072\tvalid_1's auc: 0.675393\n",
      "[169]\ttraining's auc: 0.79877\tvalid_1's auc: 0.675341\n",
      "[170]\ttraining's auc: 0.799108\tvalid_1's auc: 0.675308\n",
      "[171]\ttraining's auc: 0.799536\tvalid_1's auc: 0.675373\n",
      "[172]\ttraining's auc: 0.799652\tvalid_1's auc: 0.67541\n",
      "[173]\ttraining's auc: 0.799952\tvalid_1's auc: 0.675492\n",
      "[174]\ttraining's auc: 0.8003\tvalid_1's auc: 0.675609\n",
      "[175]\ttraining's auc: 0.800807\tvalid_1's auc: 0.675636\n",
      "[176]\ttraining's auc: 0.801381\tvalid_1's auc: 0.675736\n",
      "[177]\ttraining's auc: 0.80166\tvalid_1's auc: 0.675691\n",
      "[178]\ttraining's auc: 0.802046\tvalid_1's auc: 0.67566\n",
      "[179]\ttraining's auc: 0.80233\tvalid_1's auc: 0.675853\n",
      "[180]\ttraining's auc: 0.802729\tvalid_1's auc: 0.67583\n",
      "[181]\ttraining's auc: 0.80285\tvalid_1's auc: 0.675803\n",
      "[182]\ttraining's auc: 0.803189\tvalid_1's auc: 0.675897\n",
      "[183]\ttraining's auc: 0.803527\tvalid_1's auc: 0.675906\n",
      "[184]\ttraining's auc: 0.803776\tvalid_1's auc: 0.675775\n",
      "[185]\ttraining's auc: 0.803846\tvalid_1's auc: 0.675782\n",
      "[186]\ttraining's auc: 0.804233\tvalid_1's auc: 0.675799\n",
      "[187]\ttraining's auc: 0.804659\tvalid_1's auc: 0.675831\n",
      "[188]\ttraining's auc: 0.804797\tvalid_1's auc: 0.675831\n",
      "[189]\ttraining's auc: 0.805135\tvalid_1's auc: 0.675777\n",
      "[190]\ttraining's auc: 0.805453\tvalid_1's auc: 0.675796\n",
      "[191]\ttraining's auc: 0.805772\tvalid_1's auc: 0.675791\n",
      "[192]\ttraining's auc: 0.806216\tvalid_1's auc: 0.675794\n",
      "[193]\ttraining's auc: 0.806535\tvalid_1's auc: 0.675826\n",
      "[194]\ttraining's auc: 0.806831\tvalid_1's auc: 0.67565\n",
      "[195]\ttraining's auc: 0.806897\tvalid_1's auc: 0.675653\n",
      "[196]\ttraining's auc: 0.807279\tvalid_1's auc: 0.675611\n",
      "[197]\ttraining's auc: 0.807457\tvalid_1's auc: 0.67561\n",
      "[198]\ttraining's auc: 0.807558\tvalid_1's auc: 0.675641\n",
      "[199]\ttraining's auc: 0.807908\tvalid_1's auc: 0.675636\n",
      "[200]\ttraining's auc: 0.808113\tvalid_1's auc: 0.675603\n",
      "[201]\ttraining's auc: 0.80835\tvalid_1's auc: 0.675613\n",
      "[202]\ttraining's auc: 0.808496\tvalid_1's auc: 0.675564\n",
      "[203]\ttraining's auc: 0.808807\tvalid_1's auc: 0.675775\n",
      "[204]\ttraining's auc: 0.809125\tvalid_1's auc: 0.675714\n",
      "[205]\ttraining's auc: 0.809427\tvalid_1's auc: 0.675763\n",
      "[206]\ttraining's auc: 0.809553\tvalid_1's auc: 0.675748\n",
      "[207]\ttraining's auc: 0.809963\tvalid_1's auc: 0.675686\n",
      "[208]\ttraining's auc: 0.810188\tvalid_1's auc: 0.675801\n",
      "[209]\ttraining's auc: 0.810671\tvalid_1's auc: 0.675912\n",
      "[210]\ttraining's auc: 0.810936\tvalid_1's auc: 0.675919\n",
      "[211]\ttraining's auc: 0.811392\tvalid_1's auc: 0.675982\n",
      "[212]\ttraining's auc: 0.811459\tvalid_1's auc: 0.675903\n",
      "[213]\ttraining's auc: 0.811771\tvalid_1's auc: 0.675906\n",
      "[214]\ttraining's auc: 0.812041\tvalid_1's auc: 0.675926\n",
      "[215]\ttraining's auc: 0.812456\tvalid_1's auc: 0.675988\n",
      "[216]\ttraining's auc: 0.812775\tvalid_1's auc: 0.676038\n",
      "[217]\ttraining's auc: 0.812849\tvalid_1's auc: 0.676046\n",
      "[218]\ttraining's auc: 0.813247\tvalid_1's auc: 0.67605\n",
      "[219]\ttraining's auc: 0.813653\tvalid_1's auc: 0.676066\n",
      "[220]\ttraining's auc: 0.813954\tvalid_1's auc: 0.676064\n",
      "[221]\ttraining's auc: 0.814272\tvalid_1's auc: 0.676076\n",
      "[222]\ttraining's auc: 0.814526\tvalid_1's auc: 0.675993\n",
      "[223]\ttraining's auc: 0.814807\tvalid_1's auc: 0.676001\n",
      "[224]\ttraining's auc: 0.815267\tvalid_1's auc: 0.67598\n",
      "[225]\ttraining's auc: 0.815837\tvalid_1's auc: 0.675957\n",
      "[226]\ttraining's auc: 0.816234\tvalid_1's auc: 0.675934\n",
      "[227]\ttraining's auc: 0.816563\tvalid_1's auc: 0.675945\n",
      "[228]\ttraining's auc: 0.816768\tvalid_1's auc: 0.675922\n",
      "[229]\ttraining's auc: 0.816938\tvalid_1's auc: 0.675931\n",
      "[230]\ttraining's auc: 0.817191\tvalid_1's auc: 0.675974\n",
      "[231]\ttraining's auc: 0.817371\tvalid_1's auc: 0.675889\n",
      "[232]\ttraining's auc: 0.817671\tvalid_1's auc: 0.675862\n",
      "[233]\ttraining's auc: 0.817869\tvalid_1's auc: 0.675932\n",
      "[234]\ttraining's auc: 0.818098\tvalid_1's auc: 0.675959\n",
      "[235]\ttraining's auc: 0.818285\tvalid_1's auc: 0.675918\n",
      "[236]\ttraining's auc: 0.818599\tvalid_1's auc: 0.675964\n",
      "[237]\ttraining's auc: 0.818642\tvalid_1's auc: 0.675932\n",
      "[238]\ttraining's auc: 0.818901\tvalid_1's auc: 0.67592\n",
      "[239]\ttraining's auc: 0.818992\tvalid_1's auc: 0.675922\n",
      "[240]\ttraining's auc: 0.819182\tvalid_1's auc: 0.675967\n",
      "[241]\ttraining's auc: 0.819421\tvalid_1's auc: 0.676071\n",
      "[242]\ttraining's auc: 0.819648\tvalid_1's auc: 0.675976\n",
      "[243]\ttraining's auc: 0.819937\tvalid_1's auc: 0.676029\n",
      "[244]\ttraining's auc: 0.820231\tvalid_1's auc: 0.676007\n",
      "[245]\ttraining's auc: 0.820531\tvalid_1's auc: 0.675972\n",
      "[246]\ttraining's auc: 0.820612\tvalid_1's auc: 0.675953\n",
      "[247]\ttraining's auc: 0.820838\tvalid_1's auc: 0.675938\n",
      "[248]\ttraining's auc: 0.821191\tvalid_1's auc: 0.675966\n",
      "[249]\ttraining's auc: 0.821624\tvalid_1's auc: 0.67601\n",
      "[250]\ttraining's auc: 0.821978\tvalid_1's auc: 0.676038\n",
      "[251]\ttraining's auc: 0.822268\tvalid_1's auc: 0.676176\n",
      "[252]\ttraining's auc: 0.822605\tvalid_1's auc: 0.676221\n",
      "[253]\ttraining's auc: 0.823108\tvalid_1's auc: 0.676164\n",
      "[254]\ttraining's auc: 0.82349\tvalid_1's auc: 0.676151\n",
      "[255]\ttraining's auc: 0.823582\tvalid_1's auc: 0.676146\n",
      "[256]\ttraining's auc: 0.823863\tvalid_1's auc: 0.676029\n",
      "[257]\ttraining's auc: 0.824168\tvalid_1's auc: 0.676074\n",
      "[258]\ttraining's auc: 0.824294\tvalid_1's auc: 0.676162\n",
      "[259]\ttraining's auc: 0.824543\tvalid_1's auc: 0.676184\n",
      "[260]\ttraining's auc: 0.824663\tvalid_1's auc: 0.676145\n",
      "[261]\ttraining's auc: 0.824976\tvalid_1's auc: 0.676146\n",
      "[262]\ttraining's auc: 0.825231\tvalid_1's auc: 0.67614\n",
      "[263]\ttraining's auc: 0.825514\tvalid_1's auc: 0.676181\n",
      "[264]\ttraining's auc: 0.825575\tvalid_1's auc: 0.676186\n",
      "[265]\ttraining's auc: 0.826091\tvalid_1's auc: 0.676183\n",
      "[266]\ttraining's auc: 0.826384\tvalid_1's auc: 0.67617\n",
      "[267]\ttraining's auc: 0.826563\tvalid_1's auc: 0.676192\n",
      "[268]\ttraining's auc: 0.826757\tvalid_1's auc: 0.676106\n",
      "[269]\ttraining's auc: 0.82701\tvalid_1's auc: 0.676089\n",
      "[270]\ttraining's auc: 0.82724\tvalid_1's auc: 0.67617\n",
      "[271]\ttraining's auc: 0.827743\tvalid_1's auc: 0.676205\n",
      "[272]\ttraining's auc: 0.827866\tvalid_1's auc: 0.67617\n",
      "[273]\ttraining's auc: 0.828069\tvalid_1's auc: 0.676178\n",
      "[274]\ttraining's auc: 0.828352\tvalid_1's auc: 0.676139\n",
      "[275]\ttraining's auc: 0.828543\tvalid_1's auc: 0.676148\n",
      "[276]\ttraining's auc: 0.828749\tvalid_1's auc: 0.676234\n",
      "[277]\ttraining's auc: 0.829034\tvalid_1's auc: 0.676171\n",
      "[278]\ttraining's auc: 0.82933\tvalid_1's auc: 0.676167\n",
      "[279]\ttraining's auc: 0.82954\tvalid_1's auc: 0.676152\n",
      "[280]\ttraining's auc: 0.82989\tvalid_1's auc: 0.676174\n",
      "[281]\ttraining's auc: 0.830146\tvalid_1's auc: 0.676196\n",
      "[282]\ttraining's auc: 0.83043\tvalid_1's auc: 0.676197\n",
      "[283]\ttraining's auc: 0.83089\tvalid_1's auc: 0.676152\n",
      "[284]\ttraining's auc: 0.831446\tvalid_1's auc: 0.676115\n",
      "[285]\ttraining's auc: 0.831682\tvalid_1's auc: 0.676104\n",
      "[286]\ttraining's auc: 0.832004\tvalid_1's auc: 0.676147\n",
      "[287]\ttraining's auc: 0.832363\tvalid_1's auc: 0.6762\n",
      "[288]\ttraining's auc: 0.832629\tvalid_1's auc: 0.676224\n",
      "[289]\ttraining's auc: 0.832881\tvalid_1's auc: 0.676309\n",
      "[290]\ttraining's auc: 0.833032\tvalid_1's auc: 0.676331\n",
      "[291]\ttraining's auc: 0.833361\tvalid_1's auc: 0.676277\n",
      "[292]\ttraining's auc: 0.833655\tvalid_1's auc: 0.676311\n",
      "[293]\ttraining's auc: 0.833971\tvalid_1's auc: 0.676294\n",
      "[294]\ttraining's auc: 0.834393\tvalid_1's auc: 0.676347\n",
      "[295]\ttraining's auc: 0.834455\tvalid_1's auc: 0.676324\n",
      "[296]\ttraining's auc: 0.834696\tvalid_1's auc: 0.676369\n",
      "[297]\ttraining's auc: 0.834932\tvalid_1's auc: 0.676363\n",
      "[298]\ttraining's auc: 0.835224\tvalid_1's auc: 0.67638\n",
      "[299]\ttraining's auc: 0.835308\tvalid_1's auc: 0.676366\n",
      "[300]\ttraining's auc: 0.835511\tvalid_1's auc: 0.676379\n",
      "[301]\ttraining's auc: 0.835639\tvalid_1's auc: 0.676389\n",
      "[302]\ttraining's auc: 0.835904\tvalid_1's auc: 0.676346\n",
      "[303]\ttraining's auc: 0.836185\tvalid_1's auc: 0.676368\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[304]\ttraining's auc: 0.836358\tvalid_1's auc: 0.676398\n",
      "[305]\ttraining's auc: 0.83645\tvalid_1's auc: 0.676455\n",
      "[306]\ttraining's auc: 0.83658\tvalid_1's auc: 0.676541\n",
      "[307]\ttraining's auc: 0.836857\tvalid_1's auc: 0.676503\n",
      "[308]\ttraining's auc: 0.837038\tvalid_1's auc: 0.676554\n",
      "[309]\ttraining's auc: 0.837406\tvalid_1's auc: 0.676536\n",
      "[310]\ttraining's auc: 0.837637\tvalid_1's auc: 0.676515\n",
      "[311]\ttraining's auc: 0.837871\tvalid_1's auc: 0.676428\n",
      "[312]\ttraining's auc: 0.837993\tvalid_1's auc: 0.676398\n",
      "[313]\ttraining's auc: 0.838232\tvalid_1's auc: 0.67638\n",
      "[314]\ttraining's auc: 0.838359\tvalid_1's auc: 0.67633\n",
      "[315]\ttraining's auc: 0.838496\tvalid_1's auc: 0.676291\n",
      "[316]\ttraining's auc: 0.838676\tvalid_1's auc: 0.676284\n",
      "[317]\ttraining's auc: 0.83903\tvalid_1's auc: 0.676386\n",
      "[318]\ttraining's auc: 0.839311\tvalid_1's auc: 0.676441\n",
      "[319]\ttraining's auc: 0.839522\tvalid_1's auc: 0.676442\n",
      "[320]\ttraining's auc: 0.839726\tvalid_1's auc: 0.67645\n",
      "[321]\ttraining's auc: 0.840028\tvalid_1's auc: 0.676362\n",
      "[322]\ttraining's auc: 0.840353\tvalid_1's auc: 0.676452\n",
      "[323]\ttraining's auc: 0.84055\tvalid_1's auc: 0.676439\n",
      "[324]\ttraining's auc: 0.840829\tvalid_1's auc: 0.676343\n",
      "[325]\ttraining's auc: 0.841183\tvalid_1's auc: 0.676326\n",
      "[326]\ttraining's auc: 0.841419\tvalid_1's auc: 0.676343\n",
      "[327]\ttraining's auc: 0.84158\tvalid_1's auc: 0.676385\n",
      "[328]\ttraining's auc: 0.841718\tvalid_1's auc: 0.676387\n",
      "[329]\ttraining's auc: 0.841803\tvalid_1's auc: 0.676442\n",
      "[330]\ttraining's auc: 0.842044\tvalid_1's auc: 0.67641\n",
      "[331]\ttraining's auc: 0.842322\tvalid_1's auc: 0.676374\n",
      "[332]\ttraining's auc: 0.842567\tvalid_1's auc: 0.676336\n",
      "[333]\ttraining's auc: 0.842849\tvalid_1's auc: 0.676386\n",
      "[334]\ttraining's auc: 0.843075\tvalid_1's auc: 0.676386\n",
      "[335]\ttraining's auc: 0.843346\tvalid_1's auc: 0.676384\n",
      "[336]\ttraining's auc: 0.843495\tvalid_1's auc: 0.676398\n",
      "[337]\ttraining's auc: 0.84394\tvalid_1's auc: 0.67611\n",
      "[338]\ttraining's auc: 0.844024\tvalid_1's auc: 0.676156\n",
      "[339]\ttraining's auc: 0.844303\tvalid_1's auc: 0.676104\n",
      "[340]\ttraining's auc: 0.844495\tvalid_1's auc: 0.676124\n",
      "[341]\ttraining's auc: 0.844695\tvalid_1's auc: 0.676127\n",
      "[342]\ttraining's auc: 0.845111\tvalid_1's auc: 0.676104\n",
      "[343]\ttraining's auc: 0.845462\tvalid_1's auc: 0.676132\n",
      "[344]\ttraining's auc: 0.845756\tvalid_1's auc: 0.676164\n",
      "[345]\ttraining's auc: 0.845972\tvalid_1's auc: 0.676128\n",
      "[346]\ttraining's auc: 0.846167\tvalid_1's auc: 0.67609\n",
      "[347]\ttraining's auc: 0.84638\tvalid_1's auc: 0.676099\n",
      "[348]\ttraining's auc: 0.846503\tvalid_1's auc: 0.676092\n",
      "[349]\ttraining's auc: 0.846663\tvalid_1's auc: 0.676062\n",
      "[350]\ttraining's auc: 0.846884\tvalid_1's auc: 0.676068\n",
      "[351]\ttraining's auc: 0.846954\tvalid_1's auc: 0.676056\n",
      "[352]\ttraining's auc: 0.847286\tvalid_1's auc: 0.67618\n",
      "[353]\ttraining's auc: 0.847651\tvalid_1's auc: 0.676258\n",
      "[354]\ttraining's auc: 0.847834\tvalid_1's auc: 0.676312\n",
      "[355]\ttraining's auc: 0.847829\tvalid_1's auc: 0.676331\n",
      "[356]\ttraining's auc: 0.84818\tvalid_1's auc: 0.676341\n",
      "[357]\ttraining's auc: 0.848393\tvalid_1's auc: 0.676308\n",
      "[358]\ttraining's auc: 0.848654\tvalid_1's auc: 0.676293\n",
      "[359]\ttraining's auc: 0.848972\tvalid_1's auc: 0.676339\n",
      "[360]\ttraining's auc: 0.849056\tvalid_1's auc: 0.676339\n",
      "[361]\ttraining's auc: 0.849173\tvalid_1's auc: 0.67637\n",
      "[362]\ttraining's auc: 0.849335\tvalid_1's auc: 0.676388\n",
      "[363]\ttraining's auc: 0.849567\tvalid_1's auc: 0.676325\n",
      "[364]\ttraining's auc: 0.849927\tvalid_1's auc: 0.676339\n",
      "[365]\ttraining's auc: 0.850285\tvalid_1's auc: 0.676404\n",
      "[366]\ttraining's auc: 0.850358\tvalid_1's auc: 0.676372\n",
      "[367]\ttraining's auc: 0.850484\tvalid_1's auc: 0.676397\n",
      "[368]\ttraining's auc: 0.850682\tvalid_1's auc: 0.676452\n",
      "[369]\ttraining's auc: 0.850888\tvalid_1's auc: 0.676439\n",
      "[370]\ttraining's auc: 0.851116\tvalid_1's auc: 0.676449\n",
      "[371]\ttraining's auc: 0.851295\tvalid_1's auc: 0.676408\n",
      "[372]\ttraining's auc: 0.851617\tvalid_1's auc: 0.676446\n",
      "[373]\ttraining's auc: 0.85188\tvalid_1's auc: 0.6764\n",
      "[374]\ttraining's auc: 0.851976\tvalid_1's auc: 0.676398\n",
      "[375]\ttraining's auc: 0.852249\tvalid_1's auc: 0.67633\n",
      "[376]\ttraining's auc: 0.852561\tvalid_1's auc: 0.676305\n",
      "[377]\ttraining's auc: 0.852624\tvalid_1's auc: 0.676279\n",
      "[378]\ttraining's auc: 0.85275\tvalid_1's auc: 0.676289\n",
      "[379]\ttraining's auc: 0.85297\tvalid_1's auc: 0.676255\n",
      "[380]\ttraining's auc: 0.853116\tvalid_1's auc: 0.676238\n",
      "[381]\ttraining's auc: 0.853342\tvalid_1's auc: 0.676214\n",
      "[382]\ttraining's auc: 0.853573\tvalid_1's auc: 0.67624\n",
      "[383]\ttraining's auc: 0.853737\tvalid_1's auc: 0.676285\n",
      "[384]\ttraining's auc: 0.853849\tvalid_1's auc: 0.676304\n",
      "[385]\ttraining's auc: 0.854035\tvalid_1's auc: 0.676287\n",
      "[386]\ttraining's auc: 0.854212\tvalid_1's auc: 0.676342\n",
      "[387]\ttraining's auc: 0.854255\tvalid_1's auc: 0.676334\n",
      "[388]\ttraining's auc: 0.854493\tvalid_1's auc: 0.676304\n",
      "[389]\ttraining's auc: 0.854644\tvalid_1's auc: 0.676309\n",
      "[390]\ttraining's auc: 0.85482\tvalid_1's auc: 0.676315\n",
      "[391]\ttraining's auc: 0.855031\tvalid_1's auc: 0.676267\n",
      "[392]\ttraining's auc: 0.855363\tvalid_1's auc: 0.676199\n",
      "[393]\ttraining's auc: 0.855553\tvalid_1's auc: 0.676257\n",
      "[394]\ttraining's auc: 0.855789\tvalid_1's auc: 0.676278\n",
      "[395]\ttraining's auc: 0.855988\tvalid_1's auc: 0.676238\n",
      "[396]\ttraining's auc: 0.856196\tvalid_1's auc: 0.676154\n",
      "[397]\ttraining's auc: 0.856356\tvalid_1's auc: 0.676145\n",
      "[398]\ttraining's auc: 0.856505\tvalid_1's auc: 0.676166\n",
      "[399]\ttraining's auc: 0.856601\tvalid_1's auc: 0.676118\n",
      "[400]\ttraining's auc: 0.856874\tvalid_1's auc: 0.676162\n",
      "[401]\ttraining's auc: 0.856978\tvalid_1's auc: 0.676157\n",
      "[402]\ttraining's auc: 0.857068\tvalid_1's auc: 0.676161\n",
      "[403]\ttraining's auc: 0.857365\tvalid_1's auc: 0.676197\n",
      "[404]\ttraining's auc: 0.857467\tvalid_1's auc: 0.676187\n",
      "[405]\ttraining's auc: 0.857593\tvalid_1's auc: 0.676158\n",
      "[406]\ttraining's auc: 0.85766\tvalid_1's auc: 0.676156\n",
      "[407]\ttraining's auc: 0.857838\tvalid_1's auc: 0.676195\n",
      "[408]\ttraining's auc: 0.858109\tvalid_1's auc: 0.676207\n",
      "[409]\ttraining's auc: 0.858231\tvalid_1's auc: 0.676221\n",
      "[410]\ttraining's auc: 0.85837\tvalid_1's auc: 0.676245\n",
      "[411]\ttraining's auc: 0.85848\tvalid_1's auc: 0.676237\n",
      "[412]\ttraining's auc: 0.85864\tvalid_1's auc: 0.67624\n",
      "[413]\ttraining's auc: 0.858734\tvalid_1's auc: 0.676243\n",
      "[414]\ttraining's auc: 0.859022\tvalid_1's auc: 0.676261\n",
      "[415]\ttraining's auc: 0.859037\tvalid_1's auc: 0.676253\n",
      "[416]\ttraining's auc: 0.859408\tvalid_1's auc: 0.676177\n",
      "[417]\ttraining's auc: 0.859629\tvalid_1's auc: 0.676113\n",
      "[418]\ttraining's auc: 0.859734\tvalid_1's auc: 0.676086\n",
      "[419]\ttraining's auc: 0.859986\tvalid_1's auc: 0.6761\n",
      "[420]\ttraining's auc: 0.860271\tvalid_1's auc: 0.676096\n",
      "[421]\ttraining's auc: 0.860655\tvalid_1's auc: 0.676179\n",
      "[422]\ttraining's auc: 0.86094\tvalid_1's auc: 0.676235\n",
      "[423]\ttraining's auc: 0.861176\tvalid_1's auc: 0.676245\n",
      "[424]\ttraining's auc: 0.861515\tvalid_1's auc: 0.676248\n",
      "[425]\ttraining's auc: 0.861792\tvalid_1's auc: 0.676263\n",
      "[426]\ttraining's auc: 0.86191\tvalid_1's auc: 0.676266\n",
      "[427]\ttraining's auc: 0.862152\tvalid_1's auc: 0.676253\n",
      "[428]\ttraining's auc: 0.862357\tvalid_1's auc: 0.67629\n",
      "[429]\ttraining's auc: 0.862507\tvalid_1's auc: 0.676234\n",
      "[430]\ttraining's auc: 0.86262\tvalid_1's auc: 0.676248\n",
      "[431]\ttraining's auc: 0.862728\tvalid_1's auc: 0.676263\n",
      "[432]\ttraining's auc: 0.862961\tvalid_1's auc: 0.676273\n",
      "[433]\ttraining's auc: 0.863048\tvalid_1's auc: 0.676241\n",
      "[434]\ttraining's auc: 0.863206\tvalid_1's auc: 0.676278\n",
      "[435]\ttraining's auc: 0.863284\tvalid_1's auc: 0.676259\n",
      "[436]\ttraining's auc: 0.863505\tvalid_1's auc: 0.676249\n",
      "[437]\ttraining's auc: 0.863711\tvalid_1's auc: 0.676294\n",
      "[438]\ttraining's auc: 0.864088\tvalid_1's auc: 0.676272\n",
      "[439]\ttraining's auc: 0.86417\tvalid_1's auc: 0.676269\n",
      "[440]\ttraining's auc: 0.86433\tvalid_1's auc: 0.676281\n",
      "[441]\ttraining's auc: 0.864459\tvalid_1's auc: 0.676248\n",
      "[442]\ttraining's auc: 0.864671\tvalid_1's auc: 0.676336\n",
      "[443]\ttraining's auc: 0.864781\tvalid_1's auc: 0.676284\n",
      "[444]\ttraining's auc: 0.864916\tvalid_1's auc: 0.676336\n",
      "[445]\ttraining's auc: 0.864995\tvalid_1's auc: 0.676345\n",
      "[446]\ttraining's auc: 0.865128\tvalid_1's auc: 0.676255\n",
      "[447]\ttraining's auc: 0.865297\tvalid_1's auc: 0.676268\n",
      "[448]\ttraining's auc: 0.865662\tvalid_1's auc: 0.676319\n",
      "[449]\ttraining's auc: 0.865784\tvalid_1's auc: 0.676293\n",
      "[450]\ttraining's auc: 0.866048\tvalid_1's auc: 0.676291\n",
      "[451]\ttraining's auc: 0.866171\tvalid_1's auc: 0.676318\n",
      "[452]\ttraining's auc: 0.86654\tvalid_1's auc: 0.676309\n",
      "[453]\ttraining's auc: 0.866758\tvalid_1's auc: 0.676309\n",
      "[454]\ttraining's auc: 0.866849\tvalid_1's auc: 0.67631\n",
      "[455]\ttraining's auc: 0.86689\tvalid_1's auc: 0.676319\n",
      "[456]\ttraining's auc: 0.866951\tvalid_1's auc: 0.676302\n",
      "[457]\ttraining's auc: 0.867104\tvalid_1's auc: 0.676297\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[458]\ttraining's auc: 0.867299\tvalid_1's auc: 0.676295\n",
      "[459]\ttraining's auc: 0.867513\tvalid_1's auc: 0.676271\n",
      "[460]\ttraining's auc: 0.867669\tvalid_1's auc: 0.676243\n",
      "[461]\ttraining's auc: 0.867871\tvalid_1's auc: 0.676233\n",
      "[462]\ttraining's auc: 0.868058\tvalid_1's auc: 0.67625\n",
      "[463]\ttraining's auc: 0.868158\tvalid_1's auc: 0.676284\n",
      "[464]\ttraining's auc: 0.868357\tvalid_1's auc: 0.67629\n",
      "[465]\ttraining's auc: 0.868459\tvalid_1's auc: 0.676286\n",
      "[466]\ttraining's auc: 0.868663\tvalid_1's auc: 0.676333\n",
      "[467]\ttraining's auc: 0.868967\tvalid_1's auc: 0.676324\n",
      "[468]\ttraining's auc: 0.869075\tvalid_1's auc: 0.676315\n",
      "[469]\ttraining's auc: 0.869303\tvalid_1's auc: 0.676338\n",
      "[470]\ttraining's auc: 0.869479\tvalid_1's auc: 0.676325\n",
      "[471]\ttraining's auc: 0.869716\tvalid_1's auc: 0.676374\n",
      "[472]\ttraining's auc: 0.870054\tvalid_1's auc: 0.676351\n",
      "[473]\ttraining's auc: 0.87014\tvalid_1's auc: 0.676348\n",
      "[474]\ttraining's auc: 0.870416\tvalid_1's auc: 0.676358\n",
      "[475]\ttraining's auc: 0.870602\tvalid_1's auc: 0.676362\n",
      "[476]\ttraining's auc: 0.870743\tvalid_1's auc: 0.676406\n",
      "[477]\ttraining's auc: 0.870746\tvalid_1's auc: 0.676429\n",
      "[478]\ttraining's auc: 0.871078\tvalid_1's auc: 0.676433\n",
      "[479]\ttraining's auc: 0.871147\tvalid_1's auc: 0.676416\n",
      "[480]\ttraining's auc: 0.871508\tvalid_1's auc: 0.676385\n",
      "[481]\ttraining's auc: 0.871658\tvalid_1's auc: 0.676403\n",
      "[482]\ttraining's auc: 0.871793\tvalid_1's auc: 0.676358\n",
      "[483]\ttraining's auc: 0.872074\tvalid_1's auc: 0.676245\n",
      "[484]\ttraining's auc: 0.872104\tvalid_1's auc: 0.67624\n",
      "[485]\ttraining's auc: 0.872203\tvalid_1's auc: 0.676253\n",
      "[486]\ttraining's auc: 0.872359\tvalid_1's auc: 0.676289\n",
      "[487]\ttraining's auc: 0.872569\tvalid_1's auc: 0.676246\n",
      "[488]\ttraining's auc: 0.872736\tvalid_1's auc: 0.676215\n",
      "[489]\ttraining's auc: 0.872847\tvalid_1's auc: 0.676199\n",
      "[490]\ttraining's auc: 0.873021\tvalid_1's auc: 0.676185\n",
      "[491]\ttraining's auc: 0.873282\tvalid_1's auc: 0.676169\n",
      "[492]\ttraining's auc: 0.8736\tvalid_1's auc: 0.676179\n",
      "[493]\ttraining's auc: 0.87372\tvalid_1's auc: 0.676163\n",
      "[494]\ttraining's auc: 0.873936\tvalid_1's auc: 0.67613\n",
      "[495]\ttraining's auc: 0.874088\tvalid_1's auc: 0.676164\n",
      "[496]\ttraining's auc: 0.87417\tvalid_1's auc: 0.676177\n",
      "[497]\ttraining's auc: 0.874372\tvalid_1's auc: 0.676208\n",
      "[498]\ttraining's auc: 0.874478\tvalid_1's auc: 0.676257\n",
      "[499]\ttraining's auc: 0.874642\tvalid_1's auc: 0.676239\n",
      "[500]\ttraining's auc: 0.874723\tvalid_1's auc: 0.676248\n",
      "[501]\ttraining's auc: 0.87473\tvalid_1's auc: 0.676247\n",
      "[502]\ttraining's auc: 0.874891\tvalid_1's auc: 0.67628\n",
      "[503]\ttraining's auc: 0.875046\tvalid_1's auc: 0.676221\n",
      "[504]\ttraining's auc: 0.875148\tvalid_1's auc: 0.676187\n",
      "[505]\ttraining's auc: 0.87527\tvalid_1's auc: 0.676119\n",
      "[506]\ttraining's auc: 0.875336\tvalid_1's auc: 0.676109\n",
      "[507]\ttraining's auc: 0.87574\tvalid_1's auc: 0.676098\n",
      "[508]\ttraining's auc: 0.875801\tvalid_1's auc: 0.676126\n",
      "Early stopping, best iteration is:\n",
      "[308]\ttraining's auc: 0.837038\tvalid_1's auc: 0.676554\n",
      "模型在测试集上的效果是0.67228。\n"
     ]
    }
   ],
   "source": [
    "model_1 = run_lgb(df_train_1, df_validation_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用人工划分验证集的方式来评估模型效果：\n",
    "\n",
    "* 验证集AUC：0.676554\n",
    "* 测试集上AUC：0.67228\n",
    "* 差值：0.004\n",
    "\n",
    "验证集和测试集上的AUC非常接近，说明人工划分的验证集，更能够评估模型在测试集上的效果。\n",
    "\n",
    "这是因为人工划分的验证集，比起交叉验证的方式，和测试集更相似。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-26T03:41:10.240488Z",
     "start_time": "2019-04-26T03:41:10.232680Z"
    }
   },
   "source": [
    "### 和测试集最相似的样本作为验证集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:24:52.801394Z",
     "start_time": "2019-04-29T14:24:52.600187Z"
    }
   },
   "outputs": [],
   "source": [
    "# 提取出训练集上，样本是测试集的概率\n",
    "df_train_copy = df_train.copy()\n",
    "df_train_copy['is_test_prob'] = preds_adv[:len(df_train)]\n",
    "\n",
    "# 根据概率排序\n",
    "df_train_copy = df_train_copy.sort_values('is_test_prob').reset_index(drop=True)\n",
    "\n",
    "# 将概率最大的20%作为验证集\n",
    "df_validation_2 = df_train_copy.iloc[int(0.8 * len(df_train)):, ]\n",
    "df_train_2 = df_train_copy.iloc[:int(0.8 * len(df_train)), ]\n",
    "\n",
    "df_validation_2.drop('is_test_prob', axis=1, inplace=True)\n",
    "df_train_2.drop('is_test_prob', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:25:05.850277Z",
     "start_time": "2019-04-29T14:24:53.434942Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\ttraining's auc: 0.688213\tvalid_1's auc: 0.642285\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[2]\ttraining's auc: 0.692815\tvalid_1's auc: 0.64517\n",
      "[3]\ttraining's auc: 0.696436\tvalid_1's auc: 0.648787\n",
      "[4]\ttraining's auc: 0.698626\tvalid_1's auc: 0.651753\n",
      "[5]\ttraining's auc: 0.700533\tvalid_1's auc: 0.653136\n",
      "[6]\ttraining's auc: 0.702502\tvalid_1's auc: 0.655382\n",
      "[7]\ttraining's auc: 0.703482\tvalid_1's auc: 0.656062\n",
      "[8]\ttraining's auc: 0.705527\tvalid_1's auc: 0.656927\n",
      "[9]\ttraining's auc: 0.706809\tvalid_1's auc: 0.65691\n",
      "[10]\ttraining's auc: 0.708477\tvalid_1's auc: 0.657349\n",
      "[11]\ttraining's auc: 0.709249\tvalid_1's auc: 0.657707\n",
      "[12]\ttraining's auc: 0.710815\tvalid_1's auc: 0.658642\n",
      "[13]\ttraining's auc: 0.713306\tvalid_1's auc: 0.658696\n",
      "[14]\ttraining's auc: 0.714219\tvalid_1's auc: 0.659128\n",
      "[15]\ttraining's auc: 0.715036\tvalid_1's auc: 0.658691\n",
      "[16]\ttraining's auc: 0.715994\tvalid_1's auc: 0.659692\n",
      "[17]\ttraining's auc: 0.717335\tvalid_1's auc: 0.659841\n",
      "[18]\ttraining's auc: 0.718214\tvalid_1's auc: 0.660614\n",
      "[19]\ttraining's auc: 0.719374\tvalid_1's auc: 0.66052\n",
      "[20]\ttraining's auc: 0.719653\tvalid_1's auc: 0.660535\n",
      "[21]\ttraining's auc: 0.720745\tvalid_1's auc: 0.66094\n",
      "[22]\ttraining's auc: 0.722196\tvalid_1's auc: 0.660863\n",
      "[23]\ttraining's auc: 0.723391\tvalid_1's auc: 0.661537\n",
      "[24]\ttraining's auc: 0.724653\tvalid_1's auc: 0.661762\n",
      "[25]\ttraining's auc: 0.726225\tvalid_1's auc: 0.662221\n",
      "[26]\ttraining's auc: 0.726761\tvalid_1's auc: 0.66276\n",
      "[27]\ttraining's auc: 0.72752\tvalid_1's auc: 0.663231\n",
      "[28]\ttraining's auc: 0.729185\tvalid_1's auc: 0.663595\n",
      "[29]\ttraining's auc: 0.729901\tvalid_1's auc: 0.663649\n",
      "[30]\ttraining's auc: 0.730697\tvalid_1's auc: 0.663857\n",
      "[31]\ttraining's auc: 0.731587\tvalid_1's auc: 0.663977\n",
      "[32]\ttraining's auc: 0.732376\tvalid_1's auc: 0.663893\n",
      "[33]\ttraining's auc: 0.733067\tvalid_1's auc: 0.663659\n",
      "[34]\ttraining's auc: 0.734118\tvalid_1's auc: 0.664098\n",
      "[35]\ttraining's auc: 0.735236\tvalid_1's auc: 0.663923\n",
      "[36]\ttraining's auc: 0.735804\tvalid_1's auc: 0.663994\n",
      "[37]\ttraining's auc: 0.736301\tvalid_1's auc: 0.663787\n",
      "[38]\ttraining's auc: 0.736869\tvalid_1's auc: 0.66396\n",
      "[39]\ttraining's auc: 0.737424\tvalid_1's auc: 0.664015\n",
      "[40]\ttraining's auc: 0.73864\tvalid_1's auc: 0.66441\n",
      "[41]\ttraining's auc: 0.739186\tvalid_1's auc: 0.664582\n",
      "[42]\ttraining's auc: 0.74001\tvalid_1's auc: 0.664855\n",
      "[43]\ttraining's auc: 0.740952\tvalid_1's auc: 0.66507\n",
      "[44]\ttraining's auc: 0.741277\tvalid_1's auc: 0.665144\n",
      "[45]\ttraining's auc: 0.741607\tvalid_1's auc: 0.66524\n",
      "[46]\ttraining's auc: 0.742661\tvalid_1's auc: 0.665521\n",
      "[47]\ttraining's auc: 0.743048\tvalid_1's auc: 0.665372\n",
      "[48]\ttraining's auc: 0.743994\tvalid_1's auc: 0.665674\n",
      "[49]\ttraining's auc: 0.744485\tvalid_1's auc: 0.665659\n",
      "[50]\ttraining's auc: 0.745183\tvalid_1's auc: 0.665711\n",
      "[51]\ttraining's auc: 0.746046\tvalid_1's auc: 0.665582\n",
      "[52]\ttraining's auc: 0.74691\tvalid_1's auc: 0.665699\n",
      "[53]\ttraining's auc: 0.747667\tvalid_1's auc: 0.665605\n",
      "[54]\ttraining's auc: 0.748244\tvalid_1's auc: 0.665365\n",
      "[55]\ttraining's auc: 0.748557\tvalid_1's auc: 0.665348\n",
      "[56]\ttraining's auc: 0.749017\tvalid_1's auc: 0.665158\n",
      "[57]\ttraining's auc: 0.749773\tvalid_1's auc: 0.665259\n",
      "[58]\ttraining's auc: 0.750322\tvalid_1's auc: 0.665034\n",
      "[59]\ttraining's auc: 0.750906\tvalid_1's auc: 0.664964\n",
      "[60]\ttraining's auc: 0.751369\tvalid_1's auc: 0.664955\n",
      "[61]\ttraining's auc: 0.751971\tvalid_1's auc: 0.665216\n",
      "[62]\ttraining's auc: 0.752453\tvalid_1's auc: 0.665191\n",
      "[63]\ttraining's auc: 0.752591\tvalid_1's auc: 0.665241\n",
      "[64]\ttraining's auc: 0.753276\tvalid_1's auc: 0.66529\n",
      "[65]\ttraining's auc: 0.753887\tvalid_1's auc: 0.665223\n",
      "[66]\ttraining's auc: 0.754415\tvalid_1's auc: 0.665098\n",
      "[67]\ttraining's auc: 0.75525\tvalid_1's auc: 0.664851\n",
      "[68]\ttraining's auc: 0.755964\tvalid_1's auc: 0.664842\n",
      "[69]\ttraining's auc: 0.756701\tvalid_1's auc: 0.664747\n",
      "[70]\ttraining's auc: 0.757445\tvalid_1's auc: 0.664734\n",
      "[71]\ttraining's auc: 0.757708\tvalid_1's auc: 0.664588\n",
      "[72]\ttraining's auc: 0.758159\tvalid_1's auc: 0.664612\n",
      "[73]\ttraining's auc: 0.758671\tvalid_1's auc: 0.664427\n",
      "[74]\ttraining's auc: 0.758981\tvalid_1's auc: 0.664438\n",
      "[75]\ttraining's auc: 0.759565\tvalid_1's auc: 0.66457\n",
      "[76]\ttraining's auc: 0.760013\tvalid_1's auc: 0.664634\n",
      "[77]\ttraining's auc: 0.760526\tvalid_1's auc: 0.66453\n",
      "[78]\ttraining's auc: 0.761044\tvalid_1's auc: 0.66442\n",
      "[79]\ttraining's auc: 0.761626\tvalid_1's auc: 0.664388\n",
      "[80]\ttraining's auc: 0.761951\tvalid_1's auc: 0.664508\n",
      "[81]\ttraining's auc: 0.762132\tvalid_1's auc: 0.664361\n",
      "[82]\ttraining's auc: 0.762727\tvalid_1's auc: 0.664214\n",
      "[83]\ttraining's auc: 0.763103\tvalid_1's auc: 0.66406\n",
      "[84]\ttraining's auc: 0.76361\tvalid_1's auc: 0.663978\n",
      "[85]\ttraining's auc: 0.763919\tvalid_1's auc: 0.663967\n",
      "[86]\ttraining's auc: 0.764692\tvalid_1's auc: 0.663978\n",
      "[87]\ttraining's auc: 0.765183\tvalid_1's auc: 0.663876\n",
      "[88]\ttraining's auc: 0.765987\tvalid_1's auc: 0.664021\n",
      "[89]\ttraining's auc: 0.766595\tvalid_1's auc: 0.663933\n",
      "[90]\ttraining's auc: 0.766981\tvalid_1's auc: 0.663707\n",
      "[91]\ttraining's auc: 0.767409\tvalid_1's auc: 0.663694\n",
      "[92]\ttraining's auc: 0.768238\tvalid_1's auc: 0.663667\n",
      "[93]\ttraining's auc: 0.76865\tvalid_1's auc: 0.663573\n",
      "[94]\ttraining's auc: 0.769181\tvalid_1's auc: 0.663715\n",
      "[95]\ttraining's auc: 0.770045\tvalid_1's auc: 0.663782\n",
      "[96]\ttraining's auc: 0.770404\tvalid_1's auc: 0.663831\n",
      "[97]\ttraining's auc: 0.770965\tvalid_1's auc: 0.663796\n",
      "[98]\ttraining's auc: 0.771383\tvalid_1's auc: 0.663886\n",
      "[99]\ttraining's auc: 0.77185\tvalid_1's auc: 0.663875\n",
      "[100]\ttraining's auc: 0.772013\tvalid_1's auc: 0.663872\n",
      "[101]\ttraining's auc: 0.772518\tvalid_1's auc: 0.663745\n",
      "[102]\ttraining's auc: 0.772814\tvalid_1's auc: 0.663784\n",
      "[103]\ttraining's auc: 0.773029\tvalid_1's auc: 0.663697\n",
      "[104]\ttraining's auc: 0.773417\tvalid_1's auc: 0.663816\n",
      "[105]\ttraining's auc: 0.773718\tvalid_1's auc: 0.663783\n",
      "[106]\ttraining's auc: 0.77417\tvalid_1's auc: 0.66393\n",
      "[107]\ttraining's auc: 0.77452\tvalid_1's auc: 0.663787\n",
      "[108]\ttraining's auc: 0.774891\tvalid_1's auc: 0.663794\n",
      "[109]\ttraining's auc: 0.775182\tvalid_1's auc: 0.663754\n",
      "[110]\ttraining's auc: 0.775703\tvalid_1's auc: 0.663716\n",
      "[111]\ttraining's auc: 0.776352\tvalid_1's auc: 0.663657\n",
      "[112]\ttraining's auc: 0.776589\tvalid_1's auc: 0.663721\n",
      "[113]\ttraining's auc: 0.77684\tvalid_1's auc: 0.663858\n",
      "[114]\ttraining's auc: 0.777453\tvalid_1's auc: 0.663797\n",
      "[115]\ttraining's auc: 0.777709\tvalid_1's auc: 0.663911\n",
      "[116]\ttraining's auc: 0.778086\tvalid_1's auc: 0.663995\n",
      "[117]\ttraining's auc: 0.778176\tvalid_1's auc: 0.663917\n",
      "[118]\ttraining's auc: 0.778603\tvalid_1's auc: 0.663848\n",
      "[119]\ttraining's auc: 0.778957\tvalid_1's auc: 0.663772\n",
      "[120]\ttraining's auc: 0.779204\tvalid_1's auc: 0.663861\n",
      "[121]\ttraining's auc: 0.779569\tvalid_1's auc: 0.663701\n",
      "[122]\ttraining's auc: 0.77984\tvalid_1's auc: 0.6636\n",
      "[123]\ttraining's auc: 0.780373\tvalid_1's auc: 0.663653\n",
      "[124]\ttraining's auc: 0.780769\tvalid_1's auc: 0.663772\n",
      "[125]\ttraining's auc: 0.781311\tvalid_1's auc: 0.663712\n",
      "[126]\ttraining's auc: 0.781835\tvalid_1's auc: 0.663704\n",
      "[127]\ttraining's auc: 0.781994\tvalid_1's auc: 0.663622\n",
      "[128]\ttraining's auc: 0.782121\tvalid_1's auc: 0.663607\n",
      "[129]\ttraining's auc: 0.782375\tvalid_1's auc: 0.663479\n",
      "[130]\ttraining's auc: 0.782767\tvalid_1's auc: 0.663507\n",
      "[131]\ttraining's auc: 0.783156\tvalid_1's auc: 0.663425\n",
      "[132]\ttraining's auc: 0.783433\tvalid_1's auc: 0.663469\n",
      "[133]\ttraining's auc: 0.783777\tvalid_1's auc: 0.663441\n",
      "[134]\ttraining's auc: 0.784201\tvalid_1's auc: 0.663277\n",
      "[135]\ttraining's auc: 0.784438\tvalid_1's auc: 0.663319\n",
      "[136]\ttraining's auc: 0.784763\tvalid_1's auc: 0.663312\n",
      "[137]\ttraining's auc: 0.785104\tvalid_1's auc: 0.663342\n",
      "[138]\ttraining's auc: 0.785447\tvalid_1's auc: 0.663235\n",
      "[139]\ttraining's auc: 0.785599\tvalid_1's auc: 0.663194\n",
      "[140]\ttraining's auc: 0.786051\tvalid_1's auc: 0.663113\n",
      "[141]\ttraining's auc: 0.786333\tvalid_1's auc: 0.663101\n",
      "[142]\ttraining's auc: 0.786573\tvalid_1's auc: 0.663143\n",
      "[143]\ttraining's auc: 0.787082\tvalid_1's auc: 0.663098\n",
      "[144]\ttraining's auc: 0.787515\tvalid_1's auc: 0.663114\n",
      "[145]\ttraining's auc: 0.787854\tvalid_1's auc: 0.663\n",
      "[146]\ttraining's auc: 0.788191\tvalid_1's auc: 0.663015\n",
      "[147]\ttraining's auc: 0.788516\tvalid_1's auc: 0.662964\n",
      "[148]\ttraining's auc: 0.788989\tvalid_1's auc: 0.662937\n",
      "[149]\ttraining's auc: 0.789357\tvalid_1's auc: 0.662875\n",
      "[150]\ttraining's auc: 0.78989\tvalid_1's auc: 0.662938\n",
      "[151]\ttraining's auc: 0.790311\tvalid_1's auc: 0.662749\n",
      "[152]\ttraining's auc: 0.790606\tvalid_1's auc: 0.662946\n",
      "[153]\ttraining's auc: 0.790776\tvalid_1's auc: 0.662901\n",
      "[154]\ttraining's auc: 0.791232\tvalid_1's auc: 0.662851\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[155]\ttraining's auc: 0.791454\tvalid_1's auc: 0.662756\n",
      "[156]\ttraining's auc: 0.791628\tvalid_1's auc: 0.662718\n",
      "[157]\ttraining's auc: 0.79222\tvalid_1's auc: 0.662656\n",
      "[158]\ttraining's auc: 0.792484\tvalid_1's auc: 0.662597\n",
      "[159]\ttraining's auc: 0.792547\tvalid_1's auc: 0.662653\n",
      "[160]\ttraining's auc: 0.792786\tvalid_1's auc: 0.662752\n",
      "[161]\ttraining's auc: 0.79313\tvalid_1's auc: 0.662767\n",
      "[162]\ttraining's auc: 0.793479\tvalid_1's auc: 0.662701\n",
      "[163]\ttraining's auc: 0.793723\tvalid_1's auc: 0.662779\n",
      "[164]\ttraining's auc: 0.793951\tvalid_1's auc: 0.662746\n",
      "[165]\ttraining's auc: 0.794366\tvalid_1's auc: 0.662699\n",
      "[166]\ttraining's auc: 0.794605\tvalid_1's auc: 0.66273\n",
      "[167]\ttraining's auc: 0.794897\tvalid_1's auc: 0.662607\n",
      "[168]\ttraining's auc: 0.795341\tvalid_1's auc: 0.66278\n",
      "[169]\ttraining's auc: 0.795454\tvalid_1's auc: 0.662776\n",
      "[170]\ttraining's auc: 0.795825\tvalid_1's auc: 0.66264\n",
      "[171]\ttraining's auc: 0.796271\tvalid_1's auc: 0.662607\n",
      "[172]\ttraining's auc: 0.796526\tvalid_1's auc: 0.66259\n",
      "[173]\ttraining's auc: 0.796885\tvalid_1's auc: 0.662598\n",
      "[174]\ttraining's auc: 0.79711\tvalid_1's auc: 0.662541\n",
      "[175]\ttraining's auc: 0.797369\tvalid_1's auc: 0.662483\n",
      "[176]\ttraining's auc: 0.797547\tvalid_1's auc: 0.662515\n",
      "[177]\ttraining's auc: 0.797991\tvalid_1's auc: 0.662377\n",
      "[178]\ttraining's auc: 0.798253\tvalid_1's auc: 0.662289\n",
      "[179]\ttraining's auc: 0.798432\tvalid_1's auc: 0.662261\n",
      "[180]\ttraining's auc: 0.798779\tvalid_1's auc: 0.662104\n",
      "[181]\ttraining's auc: 0.799307\tvalid_1's auc: 0.662189\n",
      "[182]\ttraining's auc: 0.799609\tvalid_1's auc: 0.662268\n",
      "[183]\ttraining's auc: 0.80014\tvalid_1's auc: 0.662224\n",
      "[184]\ttraining's auc: 0.800329\tvalid_1's auc: 0.662129\n",
      "[185]\ttraining's auc: 0.800655\tvalid_1's auc: 0.66219\n",
      "[186]\ttraining's auc: 0.800986\tvalid_1's auc: 0.662077\n",
      "[187]\ttraining's auc: 0.801243\tvalid_1's auc: 0.662038\n",
      "[188]\ttraining's auc: 0.801566\tvalid_1's auc: 0.661915\n",
      "[189]\ttraining's auc: 0.801784\tvalid_1's auc: 0.661887\n",
      "[190]\ttraining's auc: 0.802408\tvalid_1's auc: 0.661969\n",
      "[191]\ttraining's auc: 0.803009\tvalid_1's auc: 0.66189\n",
      "[192]\ttraining's auc: 0.803477\tvalid_1's auc: 0.661827\n",
      "[193]\ttraining's auc: 0.80363\tvalid_1's auc: 0.66184\n",
      "[194]\ttraining's auc: 0.803927\tvalid_1's auc: 0.661732\n",
      "[195]\ttraining's auc: 0.80418\tvalid_1's auc: 0.661617\n",
      "[196]\ttraining's auc: 0.804526\tvalid_1's auc: 0.661373\n",
      "[197]\ttraining's auc: 0.804942\tvalid_1's auc: 0.661375\n",
      "[198]\ttraining's auc: 0.805127\tvalid_1's auc: 0.661356\n",
      "[199]\ttraining's auc: 0.80521\tvalid_1's auc: 0.661343\n",
      "[200]\ttraining's auc: 0.805584\tvalid_1's auc: 0.661311\n",
      "[201]\ttraining's auc: 0.805825\tvalid_1's auc: 0.661333\n",
      "[202]\ttraining's auc: 0.806152\tvalid_1's auc: 0.661255\n",
      "[203]\ttraining's auc: 0.806415\tvalid_1's auc: 0.661175\n",
      "[204]\ttraining's auc: 0.806615\tvalid_1's auc: 0.661125\n",
      "[205]\ttraining's auc: 0.806759\tvalid_1's auc: 0.661135\n",
      "[206]\ttraining's auc: 0.80704\tvalid_1's auc: 0.661212\n",
      "[207]\ttraining's auc: 0.807502\tvalid_1's auc: 0.661294\n",
      "[208]\ttraining's auc: 0.807892\tvalid_1's auc: 0.661348\n",
      "[209]\ttraining's auc: 0.808232\tvalid_1's auc: 0.661382\n",
      "[210]\ttraining's auc: 0.808496\tvalid_1's auc: 0.661345\n",
      "[211]\ttraining's auc: 0.808629\tvalid_1's auc: 0.661379\n",
      "[212]\ttraining's auc: 0.809007\tvalid_1's auc: 0.661271\n",
      "[213]\ttraining's auc: 0.809195\tvalid_1's auc: 0.661168\n",
      "[214]\ttraining's auc: 0.80951\tvalid_1's auc: 0.661103\n",
      "[215]\ttraining's auc: 0.809877\tvalid_1's auc: 0.66113\n",
      "[216]\ttraining's auc: 0.810156\tvalid_1's auc: 0.660915\n",
      "[217]\ttraining's auc: 0.810153\tvalid_1's auc: 0.660956\n",
      "[218]\ttraining's auc: 0.810456\tvalid_1's auc: 0.660878\n",
      "[219]\ttraining's auc: 0.810644\tvalid_1's auc: 0.660808\n",
      "[220]\ttraining's auc: 0.810883\tvalid_1's auc: 0.660791\n",
      "[221]\ttraining's auc: 0.811249\tvalid_1's auc: 0.660655\n",
      "[222]\ttraining's auc: 0.811449\tvalid_1's auc: 0.660635\n",
      "[223]\ttraining's auc: 0.81169\tvalid_1's auc: 0.660736\n",
      "[224]\ttraining's auc: 0.812238\tvalid_1's auc: 0.660821\n",
      "[225]\ttraining's auc: 0.812275\tvalid_1's auc: 0.660831\n",
      "[226]\ttraining's auc: 0.812771\tvalid_1's auc: 0.660747\n",
      "[227]\ttraining's auc: 0.813076\tvalid_1's auc: 0.660798\n",
      "[228]\ttraining's auc: 0.813207\tvalid_1's auc: 0.660773\n",
      "[229]\ttraining's auc: 0.813452\tvalid_1's auc: 0.660852\n",
      "[230]\ttraining's auc: 0.813546\tvalid_1's auc: 0.660867\n",
      "[231]\ttraining's auc: 0.813871\tvalid_1's auc: 0.660863\n",
      "[232]\ttraining's auc: 0.814012\tvalid_1's auc: 0.660813\n",
      "[233]\ttraining's auc: 0.814448\tvalid_1's auc: 0.660658\n",
      "[234]\ttraining's auc: 0.8146\tvalid_1's auc: 0.660481\n",
      "[235]\ttraining's auc: 0.814808\tvalid_1's auc: 0.660422\n",
      "[236]\ttraining's auc: 0.815069\tvalid_1's auc: 0.660477\n",
      "[237]\ttraining's auc: 0.815255\tvalid_1's auc: 0.660417\n",
      "[238]\ttraining's auc: 0.815449\tvalid_1's auc: 0.660398\n",
      "[239]\ttraining's auc: 0.815694\tvalid_1's auc: 0.660328\n",
      "[240]\ttraining's auc: 0.816067\tvalid_1's auc: 0.660326\n",
      "[241]\ttraining's auc: 0.816304\tvalid_1's auc: 0.660367\n",
      "[242]\ttraining's auc: 0.816481\tvalid_1's auc: 0.660415\n",
      "[243]\ttraining's auc: 0.816692\tvalid_1's auc: 0.660336\n",
      "[244]\ttraining's auc: 0.816803\tvalid_1's auc: 0.660331\n",
      "[245]\ttraining's auc: 0.817288\tvalid_1's auc: 0.660418\n",
      "[246]\ttraining's auc: 0.817622\tvalid_1's auc: 0.660312\n",
      "[247]\ttraining's auc: 0.818055\tvalid_1's auc: 0.660294\n",
      "[248]\ttraining's auc: 0.818486\tvalid_1's auc: 0.66036\n",
      "[249]\ttraining's auc: 0.818696\tvalid_1's auc: 0.660379\n",
      "[250]\ttraining's auc: 0.818968\tvalid_1's auc: 0.660404\n",
      "Early stopping, best iteration is:\n",
      "[50]\ttraining's auc: 0.745183\tvalid_1's auc: 0.665711\n",
      "模型在测试集上的效果是0.66196。\n"
     ]
    }
   ],
   "source": [
    "model_2 = run_lgb(df_train_2, df_validation_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用这种方式来评估模型效果：\n",
    "\n",
    "* 验证集AUC：0.665711\n",
    "* 测试集上AUC：0.66196\n",
    "* 差值：0.003\n",
    "\n",
    "差值同样远小于交叉验证的方式。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 有权重的交叉验证 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T14:25:44.421941Z",
     "start_time": "2019-04-29T14:25:06.589672Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tcv_agg's auc: 0.651778 + 0.011999\n",
      "[2]\tcv_agg's auc: 0.660672 + 0.0155175\n",
      "[3]\tcv_agg's auc: 0.658344 + 0.0140189\n",
      "[4]\tcv_agg's auc: 0.65738 + 0.01787\n",
      "[5]\tcv_agg's auc: 0.657711 + 0.0179211\n",
      "[6]\tcv_agg's auc: 0.659267 + 0.0165729\n",
      "[7]\tcv_agg's auc: 0.657827 + 0.017638\n",
      "[8]\tcv_agg's auc: 0.657777 + 0.0181964\n",
      "[9]\tcv_agg's auc: 0.658422 + 0.0179104\n",
      "[10]\tcv_agg's auc: 0.658812 + 0.017411\n",
      "[11]\tcv_agg's auc: 0.657857 + 0.0176432\n",
      "[12]\tcv_agg's auc: 0.657417 + 0.0169003\n",
      "[13]\tcv_agg's auc: 0.655824 + 0.0178147\n",
      "[14]\tcv_agg's auc: 0.655964 + 0.0178522\n",
      "[15]\tcv_agg's auc: 0.65785 + 0.0187058\n",
      "[16]\tcv_agg's auc: 0.657498 + 0.0192713\n",
      "[17]\tcv_agg's auc: 0.657857 + 0.0194838\n",
      "[18]\tcv_agg's auc: 0.659321 + 0.0190793\n",
      "[19]\tcv_agg's auc: 0.659281 + 0.0190464\n",
      "[20]\tcv_agg's auc: 0.658585 + 0.0181283\n",
      "[21]\tcv_agg's auc: 0.657659 + 0.017343\n",
      "[22]\tcv_agg's auc: 0.657752 + 0.0179614\n",
      "[23]\tcv_agg's auc: 0.657745 + 0.0175434\n",
      "[24]\tcv_agg's auc: 0.6585 + 0.0175791\n",
      "[25]\tcv_agg's auc: 0.659771 + 0.0176137\n",
      "[26]\tcv_agg's auc: 0.659547 + 0.0173385\n",
      "[27]\tcv_agg's auc: 0.659175 + 0.0168365\n",
      "[28]\tcv_agg's auc: 0.659955 + 0.0165993\n",
      "[29]\tcv_agg's auc: 0.661249 + 0.0164676\n",
      "[30]\tcv_agg's auc: 0.662078 + 0.0161401\n",
      "[31]\tcv_agg's auc: 0.661529 + 0.0165342\n",
      "[32]\tcv_agg's auc: 0.66114 + 0.0154356\n",
      "[33]\tcv_agg's auc: 0.661491 + 0.0149816\n",
      "[34]\tcv_agg's auc: 0.662447 + 0.0145374\n",
      "[35]\tcv_agg's auc: 0.662492 + 0.0147537\n",
      "[36]\tcv_agg's auc: 0.66226 + 0.0149784\n",
      "[37]\tcv_agg's auc: 0.663264 + 0.0153443\n",
      "[38]\tcv_agg's auc: 0.662608 + 0.0152093\n",
      "[39]\tcv_agg's auc: 0.662655 + 0.0147534\n",
      "[40]\tcv_agg's auc: 0.662972 + 0.0143553\n",
      "[41]\tcv_agg's auc: 0.663743 + 0.0140982\n",
      "[42]\tcv_agg's auc: 0.663863 + 0.0147055\n",
      "[43]\tcv_agg's auc: 0.66335 + 0.0141519\n",
      "[44]\tcv_agg's auc: 0.66377 + 0.0139847\n",
      "[45]\tcv_agg's auc: 0.664429 + 0.0136645\n",
      "[46]\tcv_agg's auc: 0.664314 + 0.013029\n",
      "[47]\tcv_agg's auc: 0.664806 + 0.0132331\n",
      "[48]\tcv_agg's auc: 0.665003 + 0.0134049\n",
      "[49]\tcv_agg's auc: 0.665112 + 0.0132167\n",
      "[50]\tcv_agg's auc: 0.664654 + 0.0132597\n",
      "[51]\tcv_agg's auc: 0.664723 + 0.0130717\n",
      "[52]\tcv_agg's auc: 0.664757 + 0.0134476\n",
      "[53]\tcv_agg's auc: 0.664435 + 0.0137235\n",
      "[54]\tcv_agg's auc: 0.664817 + 0.0136047\n",
      "[55]\tcv_agg's auc: 0.664655 + 0.01392\n",
      "[56]\tcv_agg's auc: 0.664265 + 0.0138392\n",
      "[57]\tcv_agg's auc: 0.66454 + 0.0138195\n",
      "[58]\tcv_agg's auc: 0.664581 + 0.013997\n",
      "[59]\tcv_agg's auc: 0.664905 + 0.0140264\n",
      "[60]\tcv_agg's auc: 0.665082 + 0.0139013\n",
      "[61]\tcv_agg's auc: 0.665035 + 0.0144649\n",
      "[62]\tcv_agg's auc: 0.664603 + 0.0142219\n",
      "[63]\tcv_agg's auc: 0.664321 + 0.0138439\n",
      "[64]\tcv_agg's auc: 0.664306 + 0.0141502\n",
      "[65]\tcv_agg's auc: 0.664128 + 0.013254\n",
      "[66]\tcv_agg's auc: 0.663434 + 0.0130921\n",
      "[67]\tcv_agg's auc: 0.66359 + 0.0132851\n",
      "[68]\tcv_agg's auc: 0.663809 + 0.0138178\n",
      "[69]\tcv_agg's auc: 0.664178 + 0.0138554\n",
      "[70]\tcv_agg's auc: 0.664522 + 0.013292\n",
      "[71]\tcv_agg's auc: 0.66443 + 0.0139018\n",
      "[72]\tcv_agg's auc: 0.663974 + 0.013928\n",
      "[73]\tcv_agg's auc: 0.663886 + 0.0140301\n",
      "[74]\tcv_agg's auc: 0.663916 + 0.014415\n",
      "[75]\tcv_agg's auc: 0.664301 + 0.0143977\n",
      "[76]\tcv_agg's auc: 0.664366 + 0.0141603\n",
      "[77]\tcv_agg's auc: 0.664247 + 0.0138651\n",
      "[78]\tcv_agg's auc: 0.664356 + 0.0138271\n",
      "[79]\tcv_agg's auc: 0.664625 + 0.0137469\n",
      "[80]\tcv_agg's auc: 0.664424 + 0.0135839\n",
      "[81]\tcv_agg's auc: 0.66413 + 0.0133511\n",
      "[82]\tcv_agg's auc: 0.663767 + 0.0133745\n",
      "[83]\tcv_agg's auc: 0.663838 + 0.0132332\n",
      "[84]\tcv_agg's auc: 0.664002 + 0.0129791\n",
      "[85]\tcv_agg's auc: 0.663978 + 0.0126498\n",
      "[86]\tcv_agg's auc: 0.663817 + 0.0128898\n",
      "[87]\tcv_agg's auc: 0.664536 + 0.0132192\n",
      "[88]\tcv_agg's auc: 0.664418 + 0.0129999\n",
      "[89]\tcv_agg's auc: 0.664452 + 0.0129547\n",
      "[90]\tcv_agg's auc: 0.664028 + 0.0130392\n",
      "[91]\tcv_agg's auc: 0.663144 + 0.0130164\n",
      "[92]\tcv_agg's auc: 0.663237 + 0.0127922\n",
      "[93]\tcv_agg's auc: 0.663317 + 0.0128308\n",
      "[94]\tcv_agg's auc: 0.663296 + 0.0125585\n",
      "[95]\tcv_agg's auc: 0.663164 + 0.0124952\n",
      "[96]\tcv_agg's auc: 0.663334 + 0.0127997\n",
      "[97]\tcv_agg's auc: 0.663841 + 0.0126651\n",
      "[98]\tcv_agg's auc: 0.663745 + 0.0126491\n",
      "[99]\tcv_agg's auc: 0.663923 + 0.0126427\n",
      "[100]\tcv_agg's auc: 0.663662 + 0.0126583\n",
      "[101]\tcv_agg's auc: 0.663752 + 0.013166\n",
      "[102]\tcv_agg's auc: 0.664117 + 0.013264\n",
      "[103]\tcv_agg's auc: 0.664202 + 0.0129613\n",
      "[104]\tcv_agg's auc: 0.663848 + 0.0132832\n",
      "[105]\tcv_agg's auc: 0.663665 + 0.0128453\n",
      "[106]\tcv_agg's auc: 0.664066 + 0.0128939\n",
      "[107]\tcv_agg's auc: 0.66377 + 0.0127611\n",
      "[108]\tcv_agg's auc: 0.664086 + 0.0129528\n",
      "[109]\tcv_agg's auc: 0.664061 + 0.0132193\n",
      "[110]\tcv_agg's auc: 0.664306 + 0.0129051\n",
      "[111]\tcv_agg's auc: 0.664272 + 0.0129942\n",
      "[112]\tcv_agg's auc: 0.663973 + 0.0132045\n",
      "[113]\tcv_agg's auc: 0.663928 + 0.0132395\n",
      "[114]\tcv_agg's auc: 0.663852 + 0.0129778\n",
      "[115]\tcv_agg's auc: 0.664083 + 0.0128849\n",
      "[116]\tcv_agg's auc: 0.664486 + 0.0129342\n",
      "[117]\tcv_agg's auc: 0.664494 + 0.0126994\n",
      "[118]\tcv_agg's auc: 0.664149 + 0.0123771\n",
      "[119]\tcv_agg's auc: 0.664002 + 0.0117535\n",
      "[120]\tcv_agg's auc: 0.663995 + 0.0119166\n",
      "[121]\tcv_agg's auc: 0.663955 + 0.0118472\n",
      "[122]\tcv_agg's auc: 0.664153 + 0.011743\n",
      "[123]\tcv_agg's auc: 0.66418 + 0.0117718\n",
      "[124]\tcv_agg's auc: 0.66453 + 0.0117601\n",
      "[125]\tcv_agg's auc: 0.664953 + 0.0119513\n",
      "[126]\tcv_agg's auc: 0.664844 + 0.0121024\n",
      "[127]\tcv_agg's auc: 0.664657 + 0.0118537\n",
      "[128]\tcv_agg's auc: 0.664418 + 0.0116715\n",
      "[129]\tcv_agg's auc: 0.664624 + 0.011496\n",
      "[130]\tcv_agg's auc: 0.664282 + 0.0116958\n",
      "[131]\tcv_agg's auc: 0.664027 + 0.0115229\n",
      "[132]\tcv_agg's auc: 0.664137 + 0.0116315\n",
      "[133]\tcv_agg's auc: 0.664162 + 0.0115017\n",
      "[134]\tcv_agg's auc: 0.664171 + 0.0117456\n",
      "[135]\tcv_agg's auc: 0.664214 + 0.0120119\n",
      "[136]\tcv_agg's auc: 0.664155 + 0.0121896\n",
      "[137]\tcv_agg's auc: 0.663984 + 0.0120142\n",
      "[138]\tcv_agg's auc: 0.664059 + 0.0118757\n",
      "[139]\tcv_agg's auc: 0.664014 + 0.0117544\n",
      "[140]\tcv_agg's auc: 0.663973 + 0.0118451\n",
      "[141]\tcv_agg's auc: 0.664418 + 0.0120025\n",
      "[142]\tcv_agg's auc: 0.663957 + 0.0117651\n",
      "[143]\tcv_agg's auc: 0.663775 + 0.0117259\n",
      "[144]\tcv_agg's auc: 0.663792 + 0.011562\n",
      "[145]\tcv_agg's auc: 0.663837 + 0.0115734\n",
      "[146]\tcv_agg's auc: 0.66419 + 0.0114427\n",
      "[147]\tcv_agg's auc: 0.663994 + 0.0111403\n",
      "[148]\tcv_agg's auc: 0.663883 + 0.010997\n",
      "[149]\tcv_agg's auc: 0.66428 + 0.0110698\n",
      "[150]\tcv_agg's auc: 0.664324 + 0.0113662\n",
      "[151]\tcv_agg's auc: 0.664306 + 0.0115543\n",
      "[152]\tcv_agg's auc: 0.664691 + 0.0114689\n",
      "[153]\tcv_agg's auc: 0.664795 + 0.0117551\n",
      "[154]\tcv_agg's auc: 0.664772 + 0.0116389\n",
      "[155]\tcv_agg's auc: 0.664576 + 0.0117693\n",
      "[156]\tcv_agg's auc: 0.664274 + 0.0113185\n",
      "[157]\tcv_agg's auc: 0.664558 + 0.0114776\n",
      "[158]\tcv_agg's auc: 0.664383 + 0.0112009\n",
      "[159]\tcv_agg's auc: 0.664165 + 0.0112648\n",
      "[160]\tcv_agg's auc: 0.66396 + 0.0110733\n",
      "[161]\tcv_agg's auc: 0.663891 + 0.0109378\n",
      "[162]\tcv_agg's auc: 0.663809 + 0.0109196\n",
      "[163]\tcv_agg's auc: 0.664036 + 0.01074\n",
      "[164]\tcv_agg's auc: 0.663859 + 0.0109483\n",
      "[165]\tcv_agg's auc: 0.663891 + 0.0108093\n",
      "[166]\tcv_agg's auc: 0.664018 + 0.0109846\n",
      "[167]\tcv_agg's auc: 0.664052 + 0.0108137\n",
      "[168]\tcv_agg's auc: 0.663544 + 0.0107358\n",
      "[169]\tcv_agg's auc: 0.66368 + 0.0107736\n",
      "[170]\tcv_agg's auc: 0.663958 + 0.0104893\n",
      "[171]\tcv_agg's auc: 0.663644 + 0.0106998\n",
      "[172]\tcv_agg's auc: 0.663298 + 0.0106989\n",
      "[173]\tcv_agg's auc: 0.663246 + 0.0108126\n",
      "[174]\tcv_agg's auc: 0.663202 + 0.0107468\n",
      "[175]\tcv_agg's auc: 0.662998 + 0.0106274\n",
      "[176]\tcv_agg's auc: 0.662889 + 0.0103349\n",
      "[177]\tcv_agg's auc: 0.662929 + 0.0105121\n",
      "[178]\tcv_agg's auc: 0.662911 + 0.0105077\n",
      "[179]\tcv_agg's auc: 0.66265 + 0.0104792\n",
      "[180]\tcv_agg's auc: 0.662513 + 0.0105229\n",
      "[181]\tcv_agg's auc: 0.66238 + 0.0101501\n",
      "[182]\tcv_agg's auc: 0.662321 + 0.0100378\n",
      "[183]\tcv_agg's auc: 0.66241 + 0.00985505\n",
      "[184]\tcv_agg's auc: 0.662061 + 0.0100024\n",
      "[185]\tcv_agg's auc: 0.662184 + 0.0101982\n",
      "[186]\tcv_agg's auc: 0.662253 + 0.0103731\n",
      "[187]\tcv_agg's auc: 0.662267 + 0.0104303\n",
      "[188]\tcv_agg's auc: 0.662534 + 0.0100248\n",
      "[189]\tcv_agg's auc: 0.662757 + 0.0102581\n",
      "[190]\tcv_agg's auc: 0.662767 + 0.010105\n",
      "[191]\tcv_agg's auc: 0.662813 + 0.0100226\n",
      "[192]\tcv_agg's auc: 0.662637 + 0.00974528\n",
      "[193]\tcv_agg's auc: 0.662828 + 0.00989195\n",
      "[194]\tcv_agg's auc: 0.663091 + 0.0101727\n",
      "[195]\tcv_agg's auc: 0.66295 + 0.00999206\n",
      "[196]\tcv_agg's auc: 0.663059 + 0.00997231\n",
      "[197]\tcv_agg's auc: 0.662783 + 0.0101842\n",
      "[198]\tcv_agg's auc: 0.66262 + 0.00988717\n",
      "[199]\tcv_agg's auc: 0.662633 + 0.00980534\n",
      "[200]\tcv_agg's auc: 0.662609 + 0.0100568\n",
      "[201]\tcv_agg's auc: 0.662798 + 0.0100109\n",
      "[202]\tcv_agg's auc: 0.662987 + 0.0100912\n",
      "[203]\tcv_agg's auc: 0.662835 + 0.0101377\n",
      "[204]\tcv_agg's auc: 0.663032 + 0.00998678\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[205]\tcv_agg's auc: 0.662923 + 0.00980105\n",
      "[206]\tcv_agg's auc: 0.66259 + 0.00966143\n",
      "[207]\tcv_agg's auc: 0.662717 + 0.00952947\n",
      "[208]\tcv_agg's auc: 0.662515 + 0.00954792\n",
      "[209]\tcv_agg's auc: 0.662059 + 0.00951722\n",
      "[210]\tcv_agg's auc: 0.662222 + 0.00975936\n",
      "[211]\tcv_agg's auc: 0.66226 + 0.00956029\n",
      "[212]\tcv_agg's auc: 0.662343 + 0.00935288\n",
      "[213]\tcv_agg's auc: 0.662179 + 0.00939192\n",
      "[214]\tcv_agg's auc: 0.66247 + 0.00913708\n",
      "[215]\tcv_agg's auc: 0.661948 + 0.00901557\n",
      "[216]\tcv_agg's auc: 0.662004 + 0.00888755\n",
      "[217]\tcv_agg's auc: 0.661982 + 0.00871169\n",
      "[218]\tcv_agg's auc: 0.662215 + 0.00878428\n",
      "[219]\tcv_agg's auc: 0.662359 + 0.00868966\n",
      "[220]\tcv_agg's auc: 0.662152 + 0.00872557\n",
      "[221]\tcv_agg's auc: 0.661934 + 0.00863083\n",
      "[222]\tcv_agg's auc: 0.662087 + 0.00871303\n",
      "[223]\tcv_agg's auc: 0.66207 + 0.00841599\n",
      "[224]\tcv_agg's auc: 0.662374 + 0.00840024\n",
      "[225]\tcv_agg's auc: 0.662394 + 0.00847774\n",
      "[226]\tcv_agg's auc: 0.662422 + 0.00868587\n",
      "[227]\tcv_agg's auc: 0.662675 + 0.00848627\n",
      "[228]\tcv_agg's auc: 0.66231 + 0.00836845\n",
      "[229]\tcv_agg's auc: 0.662498 + 0.00854962\n",
      "[230]\tcv_agg's auc: 0.662117 + 0.00871503\n",
      "[231]\tcv_agg's auc: 0.662059 + 0.00872569\n",
      "[232]\tcv_agg's auc: 0.662091 + 0.00860058\n",
      "[233]\tcv_agg's auc: 0.662305 + 0.00845757\n",
      "[234]\tcv_agg's auc: 0.662442 + 0.00867367\n",
      "[235]\tcv_agg's auc: 0.662496 + 0.0086092\n",
      "[236]\tcv_agg's auc: 0.662538 + 0.00848122\n",
      "[237]\tcv_agg's auc: 0.662397 + 0.00846269\n",
      "[238]\tcv_agg's auc: 0.66242 + 0.00844627\n",
      "[239]\tcv_agg's auc: 0.66233 + 0.00854632\n",
      "[240]\tcv_agg's auc: 0.662278 + 0.00861636\n",
      "[241]\tcv_agg's auc: 0.662194 + 0.00847878\n",
      "[242]\tcv_agg's auc: 0.662049 + 0.00865795\n",
      "[243]\tcv_agg's auc: 0.66167 + 0.00859991\n",
      "[244]\tcv_agg's auc: 0.661692 + 0.00857928\n",
      "[245]\tcv_agg's auc: 0.661982 + 0.00858652\n",
      "[246]\tcv_agg's auc: 0.662126 + 0.0086217\n",
      "[247]\tcv_agg's auc: 0.661936 + 0.00859891\n",
      "[248]\tcv_agg's auc: 0.662137 + 0.00852071\n",
      "[249]\tcv_agg's auc: 0.662326 + 0.00849923\n",
      "交叉验证中最优的AUC为 0.66511，对应的标准差为0.01322.\n",
      "模型最优的迭代次数为49.\n",
      "模型在测试集上的效果是0.66295。\n"
     ]
    }
   ],
   "source": [
    "model_cv_wight = run_cv(df_train, sample_weight=preds_adv[:len(df_train)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用有权重的交叉验证来评估模型效果：\n",
    "\n",
    "* 有权重的交叉验证AUC：0.66511\n",
    "* 测试集上AUC：0.66295\n",
    "* 差值：0.002\n",
    "\n",
    "差值同样小于交叉验证的方式。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 对比各种方法效果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "分别使用上述提到的总共4种方法，我们来对比一下四种方法的效果，如下表："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-26T14:33:04.671619Z",
     "start_time": "2019-04-26T14:33:04.667834Z"
    }
   },
   "source": [
    "![对比表格](./images/Validation_Chart.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用交叉验证时，验证集AUC和测试集AUC的差值是最大的，远高于其他方式。说明在样本分布发生变化时，交叉验证不能够准确评估模型在测试集上的效果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 为什么评价方式是差值，而不是测试集AUC？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-27T16:19:16.386378Z",
     "start_time": "2019-04-27T16:19:16.377511Z"
    }
   },
   "source": [
    "有人可能会提到，哪种方法在测试集上的AUC最高，哪种方法就更好，不是吗？\n",
    "\n",
    "需要注意的是，本文讨论的不是“提升”模型效果的方法，而是“评估”模型效果的方法。\n",
    "\n",
    "具体来说，虽然目前看来，比如交叉验证在测试集上的AUC，略高于有权重的交叉验证。\n",
    "\n",
    "但是，当前的模型只是一个很基础的模型（Baseline Model），没有做任何的变量筛选，特征工程，以及模型调参。\n",
    "\n",
    "由于所有的优化模型的决定，都将基于验证集，而交叉验证无法准确评估模型在测试集上的效果，这将导致很多优化模型的决定是错误的。\n",
    "\n",
    "只有在有一个可靠的验证集的情况下，提升模型在验证集上效果的方法，我们才有信心认为，它也可以提升在测试集上的表现。\n",
    "\n",
    "另外，从本次比赛的结果，我们也可以发现，最终排名很好的参赛者，都没有使用交叉验证。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-28T06:22:24.992359Z",
     "start_time": "2019-04-28T06:22:24.985119Z"
    }
   },
   "source": [
    "# 结论"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-29T06:05:46.790225Z",
     "start_time": "2019-04-29T06:05:46.785341Z"
    }
   },
   "source": [
    "在样本分布发生变化时，交叉验证不能够准确评估模型在测试集上的效果。\n",
    "\n",
    "这里建议采用其他方式：\n",
    "\n",
    "* 人工划分验证集\n",
    "* 和测试集最相似的样本作为验证集\n",
    "* 有权重的交叉验证\n",
    "\n",
    "如果你有任何疑问或者建议，欢迎通过“机器学习小站”公众号留言，或者qiuyan.liu918@gmail.com联系我。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "397.92px",
    "left": "142px",
    "top": "111.12px",
    "width": "165px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "position": {
    "height": "284.12px",
    "left": "724.8px",
    "right": "20px",
    "top": "98px",
    "width": "457.72px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
